Heatmaps in OpenLayers

Some month ago I found nice post on how to create heatmaps using HTML5 canvas element. Its author, Patrick Wied, created a nice JavaScript code responsible to create a heatmap over an image depending on the mouse click or mouse move events.

»
Author's profile picture acanimal

XCode4 and the mystery with Route-Me code sense

If you follow this blog you will know I prefer to write "long" posts with nice content :) form time to time than write lots of short posts without much substance. For that I use Twitter :p

But always there an exception... and here is mine.

»
Author's profile picture acanimal

SimplyWrite, a free web distraction writing tool based on IndexedDB

NOTE: This is post is obsolete see SimplyWrite, a free web distraction writing tool.
»
Author's profile picture acanimal

acanimal is using github to share code !!!

After some time as a pending task, finally I have  created a github account to share code and any other things with you.

»
Author's profile picture acanimal

How to Create a Cross-Platform Application with NASA WorldWind & NetBeans Platform

This post is a mini tutorial on how to create an application based on NetBeans Platform that uses the WorldWind Java virtual globe.

Before to continue, you need to have some knowledge about NetBeans Platform, WWJ and JOGL.

»
Author's profile picture acanimal

Downloading files from AEMET FTP server with Java and Apache Commons Net

Some time ago the AEMET, the Spanish government meteorological agency, release many of its data publicly: weather radar, weather stations, lightnings, ... The way they do is simple and effective: public the contents on a FTP server and update on regular intervals of time. Talking with a colleague we decide to create (on our very few free time) a simple web page that allows people to see the current values of the weather stations. A map of Spain with some dots representing weather stations and a graph with showing data from the selected station.

The data

The kind of data we are looking for is stored in the '/datos_observacion/observaciones_diezminutales' folder of the FTP sever. Within it we can found two types of folders, ended with 'diezminutales' or 'estaciones'. We will work with the first one. For every day we will found a 'YYYYMMDD_diezminutales' folder which contains the day data specify by YYYYMMDD. Inside it new files are created names 'YYYYMMDDHHmm_datos.csv.gz' which contains all the weather stations data for the 10minutes interval. We also need to take into account one more consideration. A file is not immutable, that means you can read a file with some data (or file  size) and in 1 hour it can have more new data (bigger file size) because some new data has recovery and corresponds to that interval.

The design

The idea is to create some simple program that given a date (YYYYMMDD) synchronizes the corresponding remote  'YYYYMMDD_diezminutales' folder with a local one. This way we can maintain locally and update copy of the files. Once files are locally stored a different process, that means a different post :), is responsible to handle data in the right way to be used by the web application. The summary of steps are:

  1. Given a YYYYMMDD check if the remote  'YYYYMMDD_diezminutales' folder exists. If so, then synchronize it locally:
    1. Create a local 'YYYYMMDD_diezminutales' folder if not exists.
    2. For every file on remote folder:
      1. Download if it doesn't exists or his size has changes.
      2. Otherwise, ignore if it.

The implementation

Because I'm more experienced with Java I have used it to code a little program to do the previously commented synchronization. To make FTP request I have used the Apache Commons Net project. From its web page we can read:

Apache Commons Net implements the client side of many basic Internet protocols.

The Main class

This is the starter point. User must execute the program specifying a year-month-day to check YYYY-MM-DD, something like:

java -jar "aemet_v1.jar" 2011-01-25

Lets talk a bit about the main program. The first to do is to ensure user has specified a valid date:

if (args.length != 1) {
    Logger.getLogger(Main.class.getName()).log(Level.SEVERE, "You must supply once argument like YYYY-MM-DD.");
    System.exit(1);
}
SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd");
Date date = sdf.parse(args[0]);

Second, we create the folder structure which will hold the local files copy:

File localFolder = new File("./localDownloads");
localFolder.mkdir();

File localFolderObservacions = new File(localFolder, "observaciones_diezminutales");
localFolderObservacions.mkdir();

Finally, we get the files through the use of  'AdquisicioObservaciones' class:

AdquisicioObservaciones adq = new AdquisicioObservaciones(localFolderObservacions);
adq.get(date);

Putting all together:

public static void main(String[] args) {
    try {
        if (args.length != 1) {
            Logger.getLogger(Main.class.getName()).log(Level.SEVERE, "You must supply once argument like YYYY-MM-DD.");
            System.exit(1);
        }
        SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd");
        Date date = sdf.parse(args[0]);

        File localFolder = new File("./localDownloads");
        localFolder.mkdir();

        File localFolderObservacions = new File(localFolder, "observaciones_diezminutales");
        localFolderObservacions.mkdir();

        Calendar c = Utils.getCurrentCalendarUTC();
        c.setTime(date);

        AdquisicioObservaciones adq = new AdquisicioObservaciones(localFolderObservacions);
        adq.get(date);

    } catch (ParseException ex) {
        Logger.getLogger(Main.class.getName()).log(Level.SEVERE, ex.getMessage());
    }
}

The Utils class

This is a helper class with some methods used around the code. Mainly it helps creating and setting dates in UTC (because this is the format used by AEMET) and also contains a method that uncompress GZIP files:

public static boolean uncompressGzFile(File localfile, File targetlocalfile) {
    GZIPInputStream in = null;
    OutputStream out = null;
    try {
        in = new GZIPInputStream(new FileInputStream(localfile));
        out = new FileOutputStream(targetlocalfile);
        // Transfer bytes from the compressed file to the output file
        byte[] buf = new byte[1024];
        int len;
        while ((len = in.read(buf)) > 0) {
            out.write(buf, 0, len);
        }
        return true;
    } catch (IOException ex) {
        logger.severe(ex.getMessage());
        logger.log(Level.SEVERE, "There was a problem while uncompressing file ''{0}'' to ''{1}''. Exception message ''{2}''.",
                new Object[]{localfile.getName(), targetlocalfile.getName(), ex.getMessage()});
        return false;
    } finally {
        // Close the file and stream
        if (in != null) {
            try {
                in.close();
            } catch (IOException ex) {
            }
        }
        if (out != null) {
            try {
                out.close();
            } catch (IOException ex) {
            }
        }
    }
}

The AdquisicioObservaciones class

In class constructor we must specify the local folder were files must be synchronized. Later, we can execute the 'get' method as many times we want and for any day we desire. The 'get' method, as we commented,  needs a date to work:

public void get(Date date) {
    this.date = date;
    logger.log(Level.INFO, "{0} - Start: {1}", new Object[]{AdquisicioObservaciones.class.getName(), Utils.getCurrentFormattedDate()});
    handleFiles();
    logger.log(Level.INFO, "{0} - End: {1}", new Object[]{AdquisicioObservaciones.class.getName(), Utils.getCurrentFormattedDate()});
}

Mainly it stores the date on a class attribute, prints some log information and delegates the hard work to the 'handleFiles' method. First thing this method does is to create a FTP connection and change the remote working directory to 'datos_observacion/observaciones_diezminutales' and within it change to the desired day:

ftpclient = new FTPClient();

// Connect to server
ftpclient.connect(server);
ftpclient.setFileTransferMode(FTPClient.BINARY_FILE_TYPE);

// Loggin
if (!ftpclient.login("anonymous", null)) {
    logger.severe("Can't log into FTP");
    return;
}
// Change directory
if (!ftpclient.changeWorkingDirectory(folder)) {
    logger.log(Level.SEVERE, "Can''t change to folder ''{0}''.", folder);
    return;
}
// Change to day directory
String remoteDayFolder = Utils.getStringFromDate(this.date) + "_diezminutales";
if (!ftpclient.changeWorkingDirectory(remoteDayFolder)) {
    logger.log(Level.SEVERE, "Can''t change to day folder ''{0}''.", remoteDayFolder);
    return;
}

Note we set the transfer mode to binary with:

ftpclient.setFileTransferMode(FTPClient.BINARY_FILE_TYPE);

which is important because files are GZIP compressed. Then if local folder for the specified day doesn't exists it creates one:

 

// Create local directori for the day.
String dayFolder = Utils.getStringFromDate(this.date);
File folderDay = new File(this.localFolder, dayFolder);
if (!folderDay.exists()) {
    if (!folderDay.mkdir()) {
        logger.log(Level.SEVERE, "Can''t create the daily folder ''{0}''", folderDay.getAbsolutePath());
        return;
    }
}

Finally the big part comes here. The code gets the list of remote files and check which one doesn't exist locally or has changed it size:

FTPFile[] files = ftpclient.listFiles();
for (int i = 0; i < files.length; i++) {
    FTPFile ftpfile = files[i];
    long size = ftpfile.getSize();
    File localfile = new File(folderDay, ftpfile.getName());
    boolean mustBeRead = false;

    // Check if file is a real data file
    if (!ftpfile.getName().contains("_datos")) {
        continue;
    }

    totalFiles++;
    if (!localfile.exists()) {
        logger.log(Level.INFO, "File ''{0}'' doesn't exist locally",
                new Object[]{ftpfile.getName()});
        mustBeRead = true;
    } else if (Math.abs(localfile.length() - size) > 1) {
        // Ha vegades la diferencia del fitxer remot i el local difereixen en 1 byte pero son iguals.
        logger.log(Level.INFO, "File ''{0}'' size changed (before: {1}b, after: {2}b)",
                new Object[]{ftpfile.getName(), localfile.length(), size});
        mustBeRead = true;
    } else {
        logger.log(Level.INFO, "Ignored file ''{0}''", ftpfile.getName());
        totalIgnored++;
    }

    // If we need to read the file then control if any error occurs.
    if (mustBeRead) {
        try {
            downloadFile(ftpfile, localfile);
            totalDownloaded++;
        } catch (IOException ex) {
            totalErrors++;
        } finally {
            mustBeRead = false;
        }
    }
}

The next important method is 'downloadFile', which requires two parameters the remote FTP file and the local file name where we want to store data. The remote files is retrieved using the FTPClient's 'retrieveFile' method and uncompressed with the helper method Utils.uncompressGzFile():

logger.log(Level.INFO, "Downloading file ''{0}'' at ''{1}''",
        new Object[]{ftpfile.getName(), Utils.getCurrentFormattedDate()});

fos = new FileOutputStream(localfile);
ftpclient.retrieveFile(ftpfile.getName(), fos);

logger.log(Level.INFO, "Downloaded finished at ''{0}'' , size:''{1} ''bytes , timestamp: ''{2}''.",
        new Object[]{Utils.getCurrentFormattedDate(), ftpfile.getSize(), ftpfile.getTimestamp().getTime()});

// Uncompress file
String targetName = localfile.getName().replaceAll(".gz", "");
File targetlocalfile = new File(localfile.getParentFile(), targetName);
if (Utils.uncompressGzFile(localfile, targetlocalfile)) {
    //
    // TODO - Here you can handle the file.
} else {
    // If there is any error uncompressing file then remove files to
    // ensure it will be downloaded again.
    localfile.delete();
    targetlocalfile.delete();
}

Conclusions

The program is far beyond to be a serious program to maintain a database with all the weather stations data, but can you on how to download files from FTP server using Java.

More work must be done to get also the metadata information from the weather stations (name, location, etc) and merge together with the observations. Hope that will be shown in a next post.

Finally, simply to say the code is plenty of log messages which can make code cumbersome but gives lot of information about the synchronization process. A typicall output is something like:

06-mar-2011 20:46:28 org.aemetaquisition.AdquisicioObservaciones get
INFO: org.aemetaquisition.AdquisicioObservaciones - Start: 2011-03-06 20:46:28
06-mar-2011 20:46:29 org.aemetaquisition.AdquisicioObservaciones handleFiles
INFO: File '201103040000_datos.csv.gz' doesnt exist locally
06-mar-2011 20:46:29 org.aemetaquisition.AdquisicioObservaciones downloadFile
INFO: Downloading file '201103040000_datos.csv.gz' at '2011-03-06 20:46:29'
06-mar-2011 20:46:29 org.aemetaquisition.AdquisicioObservaciones downloadFile
INFO: Downloaded finished at '2011-03-06 20:46:29' , size:'13.322 'bytes , timestamp: '5/03/11 0:36'.
06-mar-2011 20:46:29 org.aemetaquisition.AdquisicioObservaciones handleFiles
...
...
...
INFO: Downloaded finished at '2011-03-06 20:47:51' , size:'13.596 'bytes , timestamp: '5/03/11 12:36'.
06-mar-2011 20:47:51 org.aemetaquisition.AdquisicioObservaciones handleFiles
INFO: Total files 144, Total downloaded 144, Total ignored 0, Total errors: 0
06-mar-2011 20:47:51 org.aemetaquisition.AdquisicioObservaciones get
INFO: org.aemetaquisition.AdquisicioObservaciones - End: 2011-03-06 20:47:51

Download

  • Source code: here. It is bundled as a NetBeans project but can you open with any text editor.
  • Binaries: here. Remember to execute the code you must set the day to check, like:
    java -jar "aemet_v1.jar" 2011-01-25

    If you execute the program for the same day, only the new files or those with size modified will be processed again.

References

»
Author's profile picture acanimal

notemarklet: Transform selected element into a sticky note

Bookmarklets are nothing new but I never created one. There are tons of available bookmarklets to make tons of useful little things but here is my little but nice bookmarklet :) called notemarklet that helps reading marking the element under the mouse and changing its looks as a sticky note. Next there are a couple of samples:

As you can see notemarker can help you reading text pages. It is a pure JavaScript code without the need of external libraries. So not expect great effects.

Next is the source code.

To use this bookmarlet simply create a new bookmark on your broser and paste the next code as URL. Looking at source code, you can learn how to get the current target element under the mouse, its position, dimensions, etc.

javascript:(function() {

    if (!window.notemarkletDiv) {
        window.notemarkletDiv = {};
    }

    var notemarklet = window.notemarklet;
	var notemarkletDiv = null;

	function createnotemarkletDiv() {
        notemarkletDiv = document.createElement('div');
        notemarkletDiv.style.position = 'absolute';
        notemarkletDiv.style.display = 'none';
        notemarkletDiv.style.border = '1px solid #DEE184';
        notemarkletDiv.style.borderRadius = '1px';
        notemarkletDiv.style.padding = '0px';
        notemarkletDiv.style.margin = '0px';
        notemarkletDiv.style.backgroundColor = '#F4F39E';
        notemarkletDiv.style.opacity = '0.4';
        notemarkletDiv.style.zIndex = 9999;
		notemarkletDiv.style.boxShadow = '3px 3px 6px #000';
		notemarkletDiv.style.webkitBoxShadow = '3px 3px 6px #000';
		notemarkletDiv.style.mozBoxShadow = '3px 3px 6px #000';
		notemarkletDiv.style.background = '-webkit-gradient(linear, 0% 0%, 0% 100%, from(#EBEB00), to(#C5C500))';
		document.getElementsByTagName('body')[0].appendChild(notemarkletDiv);
    };

	var previousTarget = null;
	function notemarkElement(event) {
		if(!notemarkletDiv) {
	    	createnotemarkletDiv();
		}
		setTimeout(function(){
			previousTarget = target;
			notemarkletDiv.style.display = 'none';

			var target = event.target;
			var targetRect = target.getBoundingClientRect();
	        notemarkletDiv.style.top = (window.pageYOffset+targetRect.top-5)+'px';
	        notemarkletDiv.style.left = (window.pageXOffset+targetRect.left-5)+'px';
	        notemarkletDiv.style.width = (targetRect.right-targetRect.left+10)+'px';
	        notemarkletDiv.style.height = (targetRect.bottom-targetRect.top+10)+'px';
		}, 200);
		setTimeout(function(){
			notemarkletDiv.style.display = 'block';
		}, 500);
	};

	document.addEventListener('mousemove', notemarkElement, false);
})();

I think code is really simple and self explanatory. Only note the use of  getBoundingClientRect function to get target position and dimension. Also, take into account, to avoid modifying the target element we create a new div element that overlaps the target.

Simple, easy and useful... anything more?

»
Author's profile picture acanimal

Working with the JavaScript XMLHttpRequest object

In the last years JavaScript has become one of the most important programming languages. Probably one of the most famous words related to JavaScript is AJAX (Asynchronous JavaScript and XML). Request from JavaScript are made through the XmlHttpRequest JavaScript object which lets you open connections, send the request and handle data returned by the server. This post is about XMLHttpRequest object, how to use it and how to handle data in a synchronous or asynchronous way.

»
Author's profile picture acanimal

JavaScript animations (tweening): Why maths are important?

In a previous post I show how to make a simple scrolling animation using jQuery. jQuery library has lots of effects, many more added in jQuery UI and much more as plugins. Of course it is fine to use existing libraries to not reinvent the wheel, but there are times you need to do an animation by your hand without the help of existing libraries and because this one needs to know the basics to implement JavaScript animations. Right, so how we can make animations in JavaScript and how we can make it smooth? Let's go.

»
Author's profile picture acanimal

How to create a preloader in Dojo

Dojo is a great framework to create RIA applications. Dojo isn't only a bunch of utility functions to select DOM nodes, change styles, make some effects, etc in contrast it is a whole framework with a great set of UI widgets.

»
Author's profile picture acanimal