A Lua Cross-Compile Web Service

There has been a lot of discussion on the NodeMCU issues list (#2315 et al) about the challenges that many Windows developers having in installing and building luac.cross.

To help Lua developers get started, I have created a simple web service to do this cross compilation for you.  Just use the following form to upload a ZIP file containing all of the Lua sources that you want to compile and then click on the REMOTE LUAC.CROSS button to return the compiled LFS image file (or the REMOTE LUAC.CROSS.INT button if you use an integer firmware build).  This will then save the compiled LFS image to your PC and you can then download this into the ESPs LFS region.  Note:

  • This remote compile now supports LFS images for both floating point and integer firmware variants.
  • If your ZIP file is called fred.zip, the image is called fred.img.
  • The root file name must conform to Lua name conventions so fred.zip in this case is fine, but usingfred#27.zip would return an error.

See my section in our online documentation on  Lua Flash Store (LFS) for more details on LFS.


Select for Zip file: 

 
     


 

Whilst this will get you started with LFS, you may prefer to run luac.cross on your host PC.  Unfortunately the sources are written assuming a POSIX rather than a Windows runtime environment.

  1. This is straightforward if you use Linux on your desktop (as I do). It took me about 3 minutes on an RPi to do the wget the dev ZIP file, explode it, do the one line change to include/user_config.h, do the make and move the luac.cross image into /usr/local/bin, and then an rm -r nodemcu-firmware-dev to clean up.
  2. Alternative if you already use our docker environment to compile your firmware then we have extended the standard scripts to support lua cross compilation.
  3. If you prefer to work locally on your PC, then you can use Windows Services for Linux (WSL) if you run Window10 Pro or Cygwin. (It took me less than 10 mins to install Cygwin and do the steps outlined in (1)  on an old laptop that can boot into Win7.)
  4. Lastly, another simple alternative is to use a RaspberryPi or the like either using PuTTY terminal sessions or a set up a webservice that you can access from a browser or PowerShell.   For those that would like to host a local webservice on ae, I have also appended the PHP action script.

PHP script to unpack ZIP file, run luac.cross and return the LFS image

<?php
ob_start(null, 0, PHP_OUTPUT_HANDLER_STDFLAGS);
print_r($_FILES); // Debug

$dbgErrLog = "/var/log/lighttpd/web-luac-cross.log";
$zipFile = $_FILES['userfile']['tmp_name'];
$upLoadName = $_FILES['userfile']['name'];
$zipName = basename($upLoadName);
$entries = [];
/*
* Clean up and exit on error
*/
function abort($error) {
  global $zipFile, $zip, $entry, $tempDir;
  if (is_uploaded_file($zipFile)) unlink($zipFile);
  if (is_resource($zip)) zip_close($zip);
  if (is_resource($entry)) zip_entry_close($entry);
  $dbgOP = ob_get_contents();
  file_put_contents($dbgErrLog, $dbgOP);
  ob_end_clean();
  $resp = htmlentities($error, ENT_QUOTES | ENT_IGNORE, "UTF-8");
  $resp = <<<EOD
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head><title>Luac.cross compile error</title></head>
<body><pre>$resp</pre></body></html>
EOD;
  http_response_code(404);
  header('Cache-Control : no-cache, no-store, must-revalidate, max-age=0');
  header('Content-type : text/html;charset=UTF-8');
  header('Content-Length: ' . strlen($resp));
  echo $resp;
  exit();
}
// Open uploaded ZIPfile, aborting on error
if (!preg_match('/^([[:alpha:]_]\w{1,32}).zip$/i', $zipName,$matches)) {
    abort("Zip file must conform to Lua variable name convention with .zip extention");
} else {
    $imageName = $matches[1].'.img';
}
if (!is_uploaded_file($zipFile)) abort("No zip file uploaded");
$zip = zip_open($zipFile);
if (!is_resource($zip)) abort("File is not a valid ZIP format");

// Unpack the uploaded ZIP file into a tmpdir
$tempDir = tempnam(sys_get_temp_dir(), 'luac_dir');
if (file_exists($tempDir)) unlink($tempDir);
mkdir($tempDir);

while (is_resource($entry = zip_read($zip))) {
  $name = basename(zip_entry_name($entry));
  $entryName = "${tempDir}/${name}";

  // Filenames must have a valid Lua name root
  if (preg_match('/^[[:alpha:]_]\w*.lua$/', $name)) {
    echo "$entryName found\n"; // Debug
    $entries[] = $entryName;
    $contents = '';
    while (($chunk = zip_entry_read($entry,8192))) {
      $contents .= $chunk;
    }
    zip_entry_close($entry); $entry = NULL;
    file_put_contents($entryName, $contents);
  }
}
zip_close($zip); $zip = NULL;

if (count($entries)==0) abort("ZIP contains no Lua files");

print_r($entries); // Debug

// Now call luac.cross to do the compile
$luaFiles = implode (" ", $entries);
$outLFS = "${tempDir}/lfs.img";
exec("luac.cross -f -o ${outLFS} ${luaFiles}", $output, $status);

if ($status != 0) abort("luac.cross compile error: \n" . implode ("\n", $output) );

$LFSimage = file_get_contents($outLFS);
exec("rm -rf ${tempDir}");

file_put_contents($dbgErrLog, ob_get_contents());
ob_end_clean();

http_response_code(200);
header('Content-Description: File Transfer');
header('Content-Type: application/octet-stream');
header("Content-Disposition: attachment; filename=\"${imageName}\"");
header('Content-Transfer-Encoding: binary');
header('Expires: 0');
header('Cache-Control: must-revalidate, no-store, must-revalidate, max-age=0');
header('Pragma: public');
header('Content-Length: ' . strlen($LFSimage));
echo $LFSimage;

This Post Has 16 Comments

  1. Hugh

    Hey Terry, firstly I found your story fascinating and it’s great to see your mind very active and your skills being applied to the IOT!! Thanks. Secondly, your work on LFS is greatly appreciated and very timely … I’ve been battling the small ram footprint of the 8266 for a while and your solution is a godsend.

    However, I’m having trouble getting an LFS image over 64KB to work. I’ve built the images using docker and have set SPIFFS_FIXED_LOCATION to 0x100000. All is good with an image < 64KB but anything larger causes the board to panic and endlessly reboot.

    I've looked around online and haven't found any tips or a solution … can you suggest a way forward?

    Thanks in advance,

    Hugh

  2. Enrique Hervás

    Hi, I have just tested LFS, and it is really a good improvment.

    I am developping an application that left me with around 6k of remaining heap, with a lot of effor to avoid the “not enough memory” and after setting the LFS, now runs with more than 35k .

    Thanks a lot for the web service, that set you up with LFS in no time.

  3. Sergey

    Today it doesn’t work for all types of builds except master int.

  4. Raffaele Diana

    Hi, i used docker but i had some problems and so i used your web service.

    The fact is, as i mentioned in github, that:
    I am having trouble getting LFS to work.

    I have created an image based on the example files https://nodemcu.readthedocs.io/en/latest/getting-started/#compile-lua-into-lfs-image and using your tool:

    https://blog.ellisons.org.uk/article/nodemcu/a-lua-cross-compile-web-service/

    I uploaded the .img file created (FLOAT MASTER but also DEV) using UPLOAD of ESplorer tool.

    After executing node.flashreload() , the firmware keeps saying that it has not loaded the lfs image (it says “No LFS image loaded”).

    Is there something i do wrong ?

    Thanks,
    Raffaele.

  5. Enrique Hervás

    I have the same problem as Raffaele: It seems that the img generated is no longer working:
    I have an image made several months ago, and works fine. It loads and works as expected.
    But if I try to build now an image from the same files, the image generated does not work. Using ESPLORER when you send node.flashreload it returns immediately, and the node reset never ocurs. Is there any change that we should be aware?
    Thanks,
    Enrique

  6. Vladimir Smirnov

    The same problem with the file *img as Raffaele Diana and Enrique Hervás. Does not work 🙁
    Thanks,
    Vladimir

  7. Vladimir Smirnov

    It is working! Terry, thank you very much for your work! This is really super!

  8. Martin

    Hi,
    I always get message “luac.cross compile error: ” when try to copile my zip. The same when I try to compile zip with the demo files only (_init and dummy_strings). There is no any error description, so I have no idea what could be wrong..

  9. eric petros

    Hi, I have worked with your LFS build service for monthes with no problem, but today I get an error on esp8266 when I run “node.flashreload(“xx.img”)” :”Incorrect LFS build type”. nothing have changed(not files, not firmware,not esp8266)

    1. Alistair Witt

      Hi Terry, I’m receiving this same error too, despite using float firmware and using the float/master button on this website. I suspect somehow it’s incorrectly generating an image for integer firmware version. Great service you’re providing however, thanks! Cheers, Alistair.

  10. Alistair Witt

    Hi Terry, I’m receiving this same error too (”Incorrect LFS build type”), despite using float firmware and using the float/master button on this website. I suspect somehow it’s incorrectly generating an image for integer firmware version. Cheers, Alistair.

  11. Daniel

    Hi, I tried your service and it works great for my project. So first of all a big thank you for all the time you spent on NodeMCU! NodeMCU memory issues led to reboot of my code within hours, now code is running for days between reboots, which is a huge improvement in everyday use. Since I am so happy I tried to build luac.cross.int on my local ubuntu 18.04 machine. Unfortunately it turned out as shared library (application/x-sharedlib) type. Took me hours to find a solution, so here it is: Add “-no-pie” to the “gcc” statement in the Makefile in app/lua/luac_cross and the application type will be correct. Finally an example for command line use: “./luac.cross -f -o my.img dummy_strings.lua _init.lua x.lua y.lua z.lua” where x,y,z, are custom modules and my.img is the resulting LFS img file

  12. Joerg

    Dear Terry, I am using your web compile service since I had the first trouble with “less memory” on my ESP. It has worked fine until today. Now the newly created .img files are not working on my ESPs anymore. So for testing I uploaded the same source file (rc8.zip) that I used so far since 05.04.2020 again but I got a different rc8.img back. So far so good unless the “old” rc8.img is still working on my ESP whereas the rc8.img from today does not. Do you have any clue what is the difference in processing the file in April and today? What do I potentially do wrong? Thank you sooo much for you help!! BR Joerg

    1. TerryE

      Sorry, but the system automatically updates the luac.cross versions every time that the dev or master is updated. You might have problems if you are running an old firmware version.

  13. Jörg Strehl

    Hi Terry, it is up&running again! Thank you very much for digging into the problem. Now I can continue with my project. 🙂 🙂 BR Joerg

  14. Arno

    Hi,

    I’m trying to use your web-service but i get a 404 not found page. Has something changed ?

    Br,

    Arno

Leave a Reply