Category Archives: Development

Full automated website experimentation with a static site generator

Posted by on July 1, 2016 No comments
Static Site Generator workflow

Lately, we’ve see a lot of static site generator tools being created by a lot of talented developers. At first sigh, you might be skeptical about the usage you could take of it. Most of the time this is advertise for a blog usage, or just a single page website. But personally, I see a lot of potential in it, you can for example create a ad listing website with a form on each ad submitted by users, you can create a full e-commerce website or you can create a full tube site. For experimentation purpose, I’ve tried to create a full tube site in full static mode + tried to automatise the content generation at it’s maximum.

What really gives all values to static sites are the side online services that could enrich drastically all the power of a single website into a full dynamic and social website.

Also, what I find really amazing with static website are the following benefits that it could bring:

  • Cost effective
  • Crazy speed
  • SEO power that they can provide!
  • (Almost) no hacks possible

How is it possible?

That’s what I’d like to demonstrate into this article as my goal was basically to generate a full tube site without having to maintain anything and put everything on autopilot. I must say that I partially succeed as there are a few features that are missing but most of the work is there yet. In order to show the full potential, I will explain the steps I went through and mention all the tools I’ve been using.

Step 1 – Choose the right static site generator

To find out the most appropriate static site generator, I’ve went through a list of mandatory features I needed. (I won’t mentioned the standard ones that are all already provided in each and everyone static site generator such as post, slug, etc…)

  • Custom collection – This is useful for creating new type of content such as in my case “videos”. We aren’t dealing with posts but with a video collection which has some different attributes (lenght, video-url, etc…)
  • Event driven plugin – This will be really interesting for fetching/scrapping automatically content from another website/API, posting automatically to social networks, minimizing, etc…
  • Sorting of collections
  • Flexible theme system
  • Command line tools

The language didn’t matter a lot but as I was more PHP fluent, I was searching for one that would fit my needs in PHP, so, I went to staticgen website to find out the best generator for my experimental website.

At first, as I’ve been reading a lot about it the time being, I’ve looked toward Sculpin. Unfortunately, what stopped me was the fact that templates weren’t well implemented yet and that custom collection (custom types) was in beta. Another criteria was that the team wasn’t maintaining the project anymore.

Then, I looked into a pretty new one that was active and proposed exactly what I wanted: Spress. I really invite you to take a look at the documentation because it’s simple & easy to read. Then take a look at a few code example on their repository and you’re good to go! Kudo guy’s for the great work!

Step 2 – Creating the template

There was no open-source template for a tube site, I needed to create my own. To speed up the process, I could have gone with a good old standard Bootstrap framework and create my blocks quickly but I wanted to learn a new CSS framework with a more UX approach. In 2015, there was a CSS framework that was making a bit of noise called Semantic-UI as they have bundle not only a lot of UI elements that all made sense but also proposed a very flexible CSS theme system where you basically just set your variables in the config files with your customization and it will change the look & feel of all your UI elements while keeping UX consistency! Same as Spress, go give it a try!

Then a problem, came to me about some dynamic values that I would need to deal with:

  • Publish date and video duration – to get over it, I’ve used the following Javascript library: MomentJS
  • Video – VideoJS is from what I know the best fitted tool for getting videos working smoothly & also responsive.
  • Social Sharing buttons – Found this on Github and works pretty fine. (I could have done this on my own but decided to go with a library that does it)

Step 3 – External services

On a tube website, you usually want users to interact with your visitors by commenting, submitting videos, making some search or simply contact us. Here are the online tools that I’ve been using:

  • Commenting – Disqus
  • Search videos – Tipusearch (a javascript search engine)
  • Video submission – Ziggeo an online service where you can simply upload or even record your video.
  • Contact form – GetSimpleForm

There is a tool that I’m missing, which is the number of page views per video. If you know any service that does just that, let me know by commenting!

Step 4 – Creating the template for Spress

After getting a fully functional template. I’ve took some examples from Spress templates and started to code my own template with all the static template I’ve been working with initially.

Step 5 – Fetching content

To fetch content & generate markdown files, I’ve used the Vid.me service which provides an API that have a bunch of information related to videos. The only thing I needed to do was the following:

  1. Retrieve from a vid.me channel all videos
  2. Download the videos & related images
  3. Generate the markdown file while injecting all the meta datas in it
  4. Run the command line for generating the website from the markdown files
  5. Put a cronjob on this script

Unfortunately, I haven’t been able to create a proper plugin for Spress with it, but it’s something that I have in my plans.

Step 6 – Wrap everything up

Once the fetcher put in place, I just needed to create a neat bash script for executing everything on a regular basis to fetch, generate and move created website to the appropriate server.

I’ve also included a plugin to minimize the HTML pages in order to speed up the delivery of the pages.

The result: Loopion Me which is a tube site dedicated to Funny Fails Videos.

Known issues:

  • Everything is currently hosted on a shared hosting called DreamHost which doesn’t allow to use enough RAM & CPU usage for generating all the pages (more than a thousands). So, either I increase my DremaHost hosting performance or Spress team needs to do efforts to reduce the amount of resources used by the page generation
  • In relation with above issue, Spress regenerate everything from start to end, which of course has the consequence to take longer to generate the full site. It would have been nice if like Hugo it was generating per update pages. They call it “partial builds”.
  • Ziggeo is providing the video through the API and I have a few submission but they aren’t published yet as I haven’t created a script to transform those incoming videos into a markdown file
  • Tags & Categories aren’t implemented

Conclusion

It was interesting to see how we can make a full website on autopilot, without any dynamic code and with a minimum security risk. Of course, the moderation is drastically missing there as you can see. But in a nutshell, the potential of delegation all dynamic work to others and not maintaining it almost at all. (I just receive a report of correct importation)

Few improvements that could bring a lot of value to your site:

  • Implement categories & tags
  • After each generated video page, send a Tweet and a Facebook post
  • Get a script to validate in one click the submitted videos on Ziggeo
  • Minimize CSS
  • Improve SEO semantics (by using Google guidelines for example)
  • Provide an authenticated (Basic Auth with SSL) API for creating native mobile apps
  • Compress images

 

Don’t hesitate to share your thoughts about this experiment. Or if you have any other suggested static site generator.

Markdown desktop editor

Posted by on March 26, 2013 1 comment
Syntax Markdown example

Markdown is really a powerful way to create documents and maintain them in a pretty way. There is no changes in the way you write from a site to another, there is no mistake done on the rendering of your pages and most of all, it’s simple to learn!

  • MarkdownPad (Windows) is a full-featured Markdown editor for Windows. There is a free and a paid version. The free version for me is way enough, but when looking to the paid services it tempting to buy the pro version.
  • Mou (Mac) is way simpler and cleaner (normal you would say “Apple style”). It does the job with no additional flavor feature included.

Both do the job and I personally use both: MarkdownPad for professional usage and Mou for personal usage.

Do you know any other Markdown apps or even offline webapps to work with Markdown documents?

Check your website links before launching

Posted by on July 10, 2012 No comments
A web spider of dots

This is often omitted by project managers but before launching the website live on the Internet, you must check that all your links from your domain point to a specific location that is reachable and (if possible) has value for your content.

Why that?

  • to make a consistent website as hyperlinks is basis of website creation.
  • have a better SEO. It will ensure a coherence for search engine spiders.
  • to not frustrate your visitor. (You know what I’m talking about)

As a consolation, if you still can’t fully be sure of your links, at least try to make as much as possible the most explainable 404 page.

To perform a full check of your links the best tool from my POV is Xenu Link Sleuth. Simple to use and easy to make reports.

Once your report build up, fix all misleading paged linked or give it away to your developers with the expected result.

I’m curious, do you have better techniques to achieve this task ?

WordPress bash deployment script

Posted by on April 2, 2011 No comments

Recently in my current company we had a client asking for a restaurant website. We’ve decided to do it with WordPress and used it as a CMS instead of a blog. Beside the fact that we needed to create the website, they also requested us to make all the maintenance of the website and their blog. What I suggested was to have a staging platform and of course the production platform.

To do make the deployment process from staging to production seamless, I’ve created a WordPress bash deployment script to easily go from staging to production. Some of you will ask me about the development platform? To do so, we’ve used a service called DeployHQ that works pretty well. It fetch all the content of your SVN repository and push it straight to the FTP. The only tricky part was for the database. I know it wasn’t clean but I made the decision to work on the same database for the staging and development part.

If you have any suggestions about improving this script do not hesitate.

Please consider the following on our installation:

#!/bin/sh
#
# Written by Emmanuel Pays 
#
# Description : Use this script to deploy in production a WordPress blog that already exist on the same machine
#
# staging_to_production_wordpress.sh ver. 1.0
#
# Latest version can be found at http://blog.loopion.com/
# Todo
# - DONE - Email when deployment is done
# - Execution time for the script
# - Set version on meta or anywhere else to keep track of deployment version

# Global config
www_path="/var/www/vhosts/${domain}/"
backup_path_db="${www_path}backup_db/"
backup_path_files="${www_path}backup_files/"
domain="yourdomain.com"
SUBJECT="${prod_uri} deployment successful"
EMAIL="your@email.com"

#Staging info
staging_uri="staging.${domain}"
root_staging_path="${www_path}subdomains/staging/httpdocs/"
staging_db_name="dbname"
staging_db_username="dbusername"
staging_db_pass="dbpass"

#Production info
prod_uri="testdeploy.${domain}"
root_prod_path="${www_path}"
prod_db_name="dbprod"
prod_db_username="dbusername"
prod_db_pass="dbpass"

#Create backup
backup_db_filename="database_$(date +%F_%Hh%M).bak.sql.bz2"
backup_file_filename="files_$(date +%F_%Hh%M).tar.bz2"

#Create files backups
tar -cjPf ${backup_path_files}prod_${backup_file_filename} ${root_prod_path}
chown ftproot:psaserv ${backup_path_files}prod_${backup_file_filename}
echo "Backup prod files done..."
tar -cjPf ${backup_path_files}staging_${backup_file_filename} ${root_staging_path}
chown ftproot:psaserv ${backup_path_files}staging_${backup_file_filename}
echo "Backup staging files done..."

mysqldump --add-drop-table -h ${staging_uri} -u ${staging_db_username} -p${staging_db_pass} ${staging_db_name} | bzip2 -c > ${backup_path_db}${backup_db_filename}
chown ftproot:psaserv ${backup_path_db}${backup_db_filename}
echo "All backups done..."

#Copy staging database to production database
mysqldump --add-drop-table -h ${staging_uri} -u ${staging_db_username} -p${staging_db_pass} ${staging_db_name} | mysql -h ${prod_uri} -u ${prod_db_username} -p${prod_db
_pass} ${prod_db_name}
echo "Database replication to production done..."
#Copy all files from staging to production
cp -rR  ${root_staging_path}. ${root_prod_path}
echo "Files copied from ${staging_uri} to ${prod_uri}"
#Update production tables
mysql -h ${prod_uri} -u ${prod_db_username} -p${prod_db_pass} ${prod_db_name}< $EMAILMESSAGE
echo "This is email text" >>$EMAILMESSAGE
# send an email using /bin/mail
mail -s "${SUBJECT}" "${EMAIL}" < $EMAILMESSAGE

Nightly backup your WordPress database and files

Posted by on March 1, 2011 No comments

After rading Using a Cron job to keep things safe I didn’t found this really a complete procedure as it only backup the database… I think it’s better to backup everything at once and we usually have some associated files like for example themes, images uploaded, etc… remaining that we don’t won’t to lost. So here is a solution to keep a copy of everything no matter what happen and unobstructive while visitors are visiting.

UPDATE 03/02/2011 -1.1:

  • Added a functionality to remove all file into the backup folder older than the specified number of days. Just set days_delete and it will delete all files older than days_delete days.
  • Added security to stop script if an error goes wrong

Just set up a cronjob and you’re good to go.

#!/bin/sh
#
# Written by Emmanuel Pays 
#
# Description : Use this script to backup a WordPress blog
#
# wordpress_backup.sh ver. 1.1
#
# Latest version can be found at http://blog.loopion.com/

# Global config
set -e
www_path="/var/www/vhosts/"
backup_path_db="${www_path}backup_db/"
backup_path_files="${www_path}backup_files/"
domain="yourdomain.tld"
days_delete=5

#Production info
prod_uri="www.${domain}"
root_prod_path="${www_path}httpdocs/"
prod_db_name="name"
prod_db_username="username"
prod_db_pass="password"
#Create backup
backup_db_filename="database_$(date +%F_%Hh%M).bak.sql.bz2"
backup_file_filename="files_$(date +%F_%Hh%M).tar.bz2"

#Create files backups
tar -cjPf ${backup_path_files}prod_${backup_file_filename} ${root_prod_path}
chown ftproot:psaserv ${backup_path_files}prod_${backup_file_filename}
echo "Backup prod files done..."

mysqldump --add-drop-table -h localhost -u ${prod_db_username} -p${prod_db_pass} ${prod_db_name} | bzip2 -c > ${backup_path_db}${backup_db_filename}
chown ftproot:psaserv ${backup_path_db}${backup_db_filename}
echo "All backups done..."

#Delete older backups
find ${backup_path_db}* -mtime +${days_delete} -exec rm {} \;
echo "Deleted last ${days_delete} database backup"
find ${backup_path_files}* -mtime +${days_delete} -exec rm {} \;
echo "Deleted last ${days_delete} files backup"

Coming soon a deployment script…

Delete all .SVN folders and files from a directory

Posted by on February 15, 2011 No comments

Nice DOS command script to remove all .svn files and folders. Some guy’s don’t know already the export function of SVN…

for /f "tokens=* delims=" %%i in ('dir /s /b /a:d *svn') do (
rd /s /q "%%i"
)