Loklak walls are going to get a new feature called manual moderation! You may have seen it in the coming soon section of the content tab when you were creating a wall.

For the walls application we have decided to use Mongodb so we can query an external API like loklak_server, then store tweets in the Mongodb database. So the first question we have to think about before storing it is how to model it in our Mongoose schemas. To answer that we have to consider the access patterns of our web app, as it is critical in deciding how we model the collection in mongodb - whether we should embed the data in an existing collection, keep all tweets for a wall in an array in one document in the collection, or store each tweet as a a document itself and query them.

The main issue is that the number of tweets could keep growing, so we can’t embed nor store them in an array in a document. Mongodb has a document size limit of 16MB (for the unfamilar, a document is to a row as collection is to a table), so we could very realistically hit that if we stored all tweets in an array.

Another issue is the frequency of access, and updates of the tweet data. If we embedded the data, we could hit the size limit and it would not make sense to keep retrieving extra information that would not be needed if we only wanted the tweets array. Also to update individual tweets we would have to save the whole array, for just one tweet!

Thus in our case I have chosen the last option, and created a separate mongodb collection to store the tweets individually as I have modeled them as a one to many, (1 walls - many tweets) relationship. Although it could also be seen as a many to many relationship between walls and tweets, to control the approval status for each requires us to treat each tweet as separate from each other as one could be approved but another rejected. So in the schema I have userWallId used for querying all tweets belonging to a certain wall of a certain user, notice the index:true option which helps speed up the query by indexing the userWallId field.

1
2
3
4
5
6
7
8

var TweetSchema = new Schema({
userWallId: { type: String, required: true, index: true },
approval: Boolean,
timestamp: String,
created_at: String,
,...
})

In contrast I have chosen to embed the wall options themselves into the user’s own collection, as this is a “one-few” relation, and we can denormalize it by embedding the wall options within an array in the user object. Denormalizing as such in Mongodb allows us to query it faster as we do not have to perform another lookup for the wall options if it were in another collection.

1
2
3
4
5
6
7
8
9
10
11
12
13
var UserSchema = new Schema({
name: { type: String, required: true },
local: {..},
isVerified: { type: Boolean, required: true },
twitter: {..},
apps: {
wall: [{
profanity: Boolean,
images: Boolean,
//...other wall options
}]
}
})

To store the tweets we will then have to create a route to handle post requests:

1
2
3
4
5
6
7
8
9
router.post ('/tweets/:userId/:wallId', auth,  function (req, res) {
req.body.forEach(function(tweet){
var newTweet = new Tweet(tweet);
newTweet.save(function(err,result){
...
});
})
res.jsonp({message: "inserted "+ req.body.length});
});

And in the front end angular app we can use these routes to store our new tweets with $http.post:

1
2
3
4
5
6
7
8
9
10
11
12
13

var posturl= '/api/tweets/'+ $rootScope.root.currentUser._id + '/' + $scope.userWalls[$scope.userWalls.length - 1].id;

SearchService.initData(searchParams).then(function(data) {

// get tweets array from loklak_server and set all tweets to default false for approval status

data.statuses.map(function(tweet){
tweet.userWallId = userWallId;
tweet.approval = false;
})

$http.post(posturl, data.statuses)

The end result is that we can do a get request at the moderation tab page for that wall’s tweets and display it as such:

Screenshot 2016-06-18 07.02.04.png

In the loklak webclient, the plan is to store the accounts locally on mongodb, instead of using loklak server to store accounts. Here’s how I used JWTs (JSON Web Tokens) for basic user authentication.

Token based authentication is often compared to the traditional method of saving a session id in a cookie, and checking the user information in the session object for every request. Instead of saving user information on the server and checking it every request, a token, which allows access to the server, is given in exchange for valid credentials and is sent in every request. There are various advantages to this, including security and scaling.

As you can see from the mongoose user schema the hash and salt are stored instead of a password.

1
2
3
4
5
6
7
8
9
10
11
12
13
var UserSchema = new Schema({
email: {
type: String,
unique: true,
required: true
},
name: {
type: String,
required: true
},
hash: String,
salt: String
});

PassportJS is a popular middleware for handling authentication in Node.js, and since the web client already uses Express, this was a natural choice. Various authentication ‘strategies’ will also be used to connect other social login providers, like twitter, or weibo.

For now, since we are using email to store the username and hashed password, we will choose the passport-local package. The configuration is easy can be found on their website, and is almost the same, except that we use email as username, rather than just any string.

1
2
3
4
5
6
7
passport.use(new LocalStrategy({
usernameField: 'email'
},
function(username, password, done) {
// ... same as example config in passportjs.org
}
})

Express.js then handles the register routes:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

router.post('/register', function(req,res){
passport.authenticate('local', function(err, user, info){
var token;
// If Passport throws/catches an error
if (err) {
res.status(404).json(err);
return;
}
// If a user is found, log in, ie. return a token
if(user){
token = user.generateJwt();
res.status(200);
res.json({
token : token
});
} else {
// If user is not found, register the user, then return a token
var user = new User();
user.name = req.body.email;
user.email = req.body.email;
user.setPassword(req.body.password);
user.save(function(err) {
var token;
token = user.generateJwt();
res.status(200);
res.json({
token : token
});
});
}
});

So that’s it for the server side code, and similar logic can be applied to the login route. Now we move on to AngularJS, where we use a service to access the above route. As you can see below the JWT is stored in local storage when the user logs in.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
function AuthenticationService($http, $window) {

var saveToken = function (token) {
$window.localStorage['jwt-token'] = token;
};

var getToken = function () {
return $window.localStorage['jwt-token'];
};

var isLoggedIn = function() {
var token = getToken();
var payload;

if(token){
payload = token.split('.')[1];
payload = $window.atob(payload); // .atob decodes the base64 String
payload = JSON.parse(payload);
return payload.exp < Date.now() / 1000;
} else {
return false;
}
};

var currentUser = function() {
if(isLoggedIn()){
var token = getToken();
var payload = token.split('.')[1];
payload = $window.atob(payload);
payload = JSON.parse(payload);
return {
email : payload.email,
name : payload.name
};
}
};

var register = function(user) {
return $http.post('/api/register', user).success(function(data){
saveToken(data.token);
});
};

The last step is to use these services, which in turn uses our routes, in our controllers:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
$rootScope.root.onSubmit = function () {
AuthService
.register($rootScope.root.credentials)
.error(function(err){
alert(err);
})
.then(function(){
$location.path('/');
$rootScope.root.isLoggedIn = AuthService.isLoggedIn();
$rootScope.root.currentUser = AuthService.currentUser();
});
};

$rootScope.root.onLogout = function () {
AuthService.logout();
$location.path('/');
$rootScope.root.isLoggedIn = AuthService.isLoggedIn();
};

And here’s what the form looks like:

3eM8YtQdYC

As you can see, there is still more work to be done to connect this with the rest of the app, but hopefully those reading can better understand how to implement local auth using JWTs on their SPA.

Loklak.org has a page for front-end apps that showcase the uses of loklak’s API, and in the first week I’ve tried to warm up for GSOC by making a ‘realtime’ twitter feed, as the existing front-end apps were mainly single http get requests and didn’t seem very appealing to me, and also to prepare for my front-end app data visualisations GSOC project.

To obtain updated data, the app polls the loklak API on a regular interval. This is done rather simply by using $interval in AngularJS, which is just a wrapper around window.setInterval(). For the realtime feed, I set the minimum rate for querying at an arbitrary 1min so as not to exceed the limits.

The next thing I needed to solve was where to store the data, as I didn’t have a database to work with, local storage was the only option. A useful module for this is “ngStorage“. At a maximum of 5MB it isn’t much, but decent enough to store a few thousand tweets and showcase the features of loklak. If developers wanted to create an external app for loklak, they could also use the front-end apps if they were well modularized.

Lastly, to display in a fancy pinterest-like grid, I’ve decided to use an angular module called “angulargrid“. To achieve the infinite scroll effect, I created another scope variable, to take a subset of the data in local storage. AngularJS then helps with rendering changes in the data model to the feed, by using $watch() on local storage.

Do checkout the tweetfeed:

tweetfeed

Some areas to improve:

  • Refactor services to make it more modular
  • Improve the regex to account for emojis and other tags

Abstract

Currently, Loklak.net has it’s own search and social wall capabilities.

alt text
alt text

Loklak.org also has a few front-end apps:

However, they are currently not as feature filled as Loklak.net, so the project aims to add additional visualisations and display options, including a live-streaming tweet wall, and timeline for loklak.org

The project

A front-end app that uses loklak.org for API calls to obtain tweets in json format. Users can deploy their own loklak server or locally or online (checkout the loklak_server github repo for one-click deploy buttons!)
Deploy
Deploy on Scalingo
Deploy to Bluemix

I have previously developed a prototype with meteor.js (github repo) which you can deploy locally or online too!

But upon discussion with the loklak team we concluded it was hard to integrate with loklak.org like the current front-end apps, so I decided to make the front-end app with a front-end framework like Angular or React.

Read more »

Profile pic

Some additions to the next theme
In _config.yml, add avatar: /images/default_avatar.jpeg

Tags page

  • Create a page named tags

    hexo new page "tags"
    
    
  • Edit tags page, set page type to tags.

    title: All tags
    date: 2014-12-22 12:39:04
    type: "tags"
    
    
  • Add tags to theme _config.yml:

    menu:
      home: /
      archives: /archives
      tags: /tags
    
    

Why Hexo?

It’s a static site generator. It’s fast, free, and simple. AND you post in markdown, which is just neat.

Creating your hexo blog 101

It’s fun & free (except for domain name, if you’re not a uni student).
First let’s setup hexo. Install npm if you haven’t.

1
2
3
4
5
6
7
npm install hexo-cli -g
hexo init
npm i # npm install
hexo s # hexo server

git clone https://github.com/iissnan/hexo-theme-next themes/next
cd themes/next/

Now go to localhost:4000, your blog should be using the default hexo theme.
We downloaded the NexT theme but have not ‘installed it’, so:

open config.yml to change theme

1
<!-- replace default "landscape" w/ "next" -->
themes: next

Now your blog should be in the next theme. hexo s to check


Create a new post/ page:

1
2
3
4
hexo new post 'blog post title' #or hexo 'post title'
hexo new page 'page title'
hexo g # generate files to /public
hexo s # start server if not already started in another terminal window

Edit it under source/_posts, check hexo’s docs for more options


Setting up domain and hosting

You must already have bought a domain, meaning the website name ‘leonmak.me’, in my case. If you have a .edu email, I recommend github education pack.

Github.io pages

Go to github.com & create organisation and within it a repo with same name (for eg using ‘leonblogs’ as the org name and repo name I will get “leonblogs/leonblogs.github.io”)
add the remote then push to the repo, in repo settings you can see link to leonblogs.github.io

Add custom domain

Create a new file in the root folder named ‘CNAME’ with just leonmak.me in it. in Domain provider (eg:Namecheap.com) Add A records: 192.30.252.153, 192.30.252.154 and CNAME: leonmak.me.

Now in settings it should link to leonmak.me

Updating with “hexo-deploy” plugin

run npm install hexo-deployer-git --save in _config.yml:

1
# Deployment
## Docs: http://hexo.io/docs/deployment.html
deploy:
  type: git
  repo: https://github.com/leonblogs/leonblogs.github.io

Add CNAME plugin for custom domain

npm install hexo-generator-cname --save

Add hexo-generator-cname to plugins in _config.yml, and set url

1
url: http://leonmak.me
<!-- ... other options -->
plugins:
- hexo-generator-cname

After editing _posts/abc.md files, do a hexo generate, to build the files, then hexo deploy.
Done!

Other plugins:

  • hexo emojis
  • Need to install hexo-util
  • Follow the instructions on the page, it’s similar to hexo-deploy