Manual Moderation - tweet storage

Loklak walls are going to get a new feature called manual moderation! You may have seen it in the coming soon section of the content tab when you were creating a wall.

For the walls application we have decided to use Mongodb so we can query an external API like loklak_server, then store tweets in the Mongodb database. So the first question we have to think about before storing it is how to model it in our Mongoose schemas. To answer that we have to consider the access patterns of our web app, as it is critical in deciding how we model the collection in mongodb - whether we should embed the data in an existing collection, keep all tweets for a wall in an array in one document in the collection, or store each tweet as a a document itself and query them.

The main issue is that the number of tweets could keep growing, so we can’t embed nor store them in an array in a document. Mongodb has a document size limit of 16MB (for the unfamilar, a document is to a row as collection is to a table), so we could very realistically hit that if we stored all tweets in an array.

Another issue is the frequency of access, and updates of the tweet data. If we embedded the data, we could hit the size limit and it would not make sense to keep retrieving extra information that would not be needed if we only wanted the tweets array. Also to update individual tweets we would have to save the whole array, for just one tweet!

Thus in our case I have chosen the last option, and created a separate mongodb collection to store the tweets individually as I have modeled them as a one to many, (1 walls - many tweets) relationship. Although it could also be seen as a many to many relationship between walls and tweets, to control the approval status for each requires us to treat each tweet as separate from each other as one could be approved but another rejected. So in the schema I have userWallId used for querying all tweets belonging to a certain wall of a certain user, notice the index:true option which helps speed up the query by indexing the userWallId field.

1
2
3
4
5
6
7
8

var TweetSchema = new Schema({
userWallId: { type: String, required: true, index: true },
approval: Boolean,
timestamp: String,
created_at: String,
,...
})

In contrast I have chosen to embed the wall options themselves into the user’s own collection, as this is a “one-few” relation, and we can denormalize it by embedding the wall options within an array in the user object. Denormalizing as such in Mongodb allows us to query it faster as we do not have to perform another lookup for the wall options if it were in another collection.

1
2
3
4
5
6
7
8
9
10
11
12
13
var UserSchema = new Schema({
name: { type: String, required: true },
local: {..},
isVerified: { type: Boolean, required: true },
twitter: {..},
apps: {
wall: [{
profanity: Boolean,
images: Boolean,
//...other wall options
}]
}
})

To store the tweets we will then have to create a route to handle post requests:

1
2
3
4
5
6
7
8
9
router.post ('/tweets/:userId/:wallId', auth,  function (req, res) {
req.body.forEach(function(tweet){
var newTweet = new Tweet(tweet);
newTweet.save(function(err,result){
...
});
})
res.jsonp({message: "inserted "+ req.body.length});
});

And in the front end angular app we can use these routes to store our new tweets with $http.post:

1
2
3
4
5
6
7
8
9
10
11
12
13

var posturl= '/api/tweets/'+ $rootScope.root.currentUser._id + '/' + $scope.userWalls[$scope.userWalls.length - 1].id;

SearchService.initData(searchParams).then(function(data) {

// get tweets array from loklak_server and set all tweets to default false for approval status

data.statuses.map(function(tweet){
tweet.userWallId = userWallId;
tweet.approval = false;
})

$http.post(posturl, data.statuses)

The end result is that we can do a get request at the moderation tab page for that wall’s tweets and display it as such:

Screenshot 2016-06-18 07.02.04.png