A Lua Cross-Compile Web Service

There has been a lot of discussion on the NodeMCU issues list (#2315 et al) about the challenges that many Windows developers having in installing and building luac.cross.

To help Lua developers get started, I have created a simple web service to do this cross compilation for you.  Just use the following form to upload a ZIP file containing all of the Lua sources that you want to compile and then click on the REMOTE LUAC.CROSS button to return the compiled LFS image file (or the REMOTE LUAC.CROSS.INT button if you use an integer firmware build).  This will then save the compiled LFS image to your PC and you can then download this into the ESPs LFS region.  Note:

  • This remote compile now supports LFS images for both floating point and integer firmware variants.
  • If your ZIP file is called fred.zip, the image is called fred.img.
  • The root file name must conform to Lua name conventions so fred.zip in this case is fine, but usingfred#27.zip would return an error.

See my section in our online documentation on  Lua Flash Store (LFS) for more details on LFS.


Select for Zip file: 

 


 

Whilst this will get you started with LFS, you may prefer to run luac.cross on your host PC.  Unfortunately the sources are written assuming a POSIX rather than a Windows runtime environment.

  1. This is straightforward if you use Linux on your desktop (as I do). It took me about 3 minutes on an RPi to do the wget the dev ZIP file, explode it, do the one line change to include/user_config.h, do the make and move the luac.cross image into /usr/local/bin, and then an rm -r nodemcu-firmware-dev to clean up.
  2. Alternative if you already use our docker environment to compile your firmware then we have extended the standard scripts to support lua cross compilation.
  3. If you prefer to work locally on your PC, then you can use Windows Services for Linux (WSL) if you run Window10 Pro or Cygwin. (It took me less than 10 mins to install Cygwin and do the steps outlined in (1)  on an old laptop that can boot into Win7.)
  4. Lastly, another simple alternative is to use a RaspberryPi or the like either using PuTTY terminal sessions or a set up a webservice that you can access from a browser or PowerShell.   For those that would like to host a local webservice on ae, I have also appended the PHP action script.

PHP script to unpack ZIP file, run luac.cross and return the LFS image

<?php
ob_start(null, 0, PHP_OUTPUT_HANDLER_STDFLAGS);
print_r($_FILES); // Debug

$dbgErrLog = "/var/log/lighttpd/web-luac-cross.log";
$zipFile = $_FILES['userfile']['tmp_name'];
$upLoadName = $_FILES['userfile']['name'];
$zipName = basename($upLoadName);
$entries = [];
/*
* Clean up and exit on error
*/
function abort($error) {
  global $zipFile, $zip, $entry, $tempDir;
  if (is_uploaded_file($zipFile)) unlink($zipFile);
  if (is_resource($zip)) zip_close($zip);
  if (is_resource($entry)) zip_entry_close($entry);
  $dbgOP = ob_get_contents();
  file_put_contents($dbgErrLog, $dbgOP);
  ob_end_clean();
  $resp = htmlentities($error, ENT_QUOTES | ENT_IGNORE, "UTF-8");
  $resp = <<<EOD
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head><title>Luac.cross compile error</title></head>
<body><pre>$resp</pre></body></html>
EOD;
  http_response_code(404);
  header('Cache-Control : no-cache, no-store, must-revalidate, max-age=0');
  header('Content-type : text/html;charset=UTF-8');
  header('Content-Length: ' . strlen($resp));
  echo $resp;
  exit();
}
// Open uploaded ZIPfile, aborting on error
if (!preg_match('/^([[:alpha:]_]\w{1,32}).zip$/i', $zipName,$matches)) {
    abort("Zip file must conform to Lua variable name convention with .zip extention");
} else {
    $imageName = $matches[1].'.img';
}
if (!is_uploaded_file($zipFile)) abort("No zip file uploaded");
$zip = zip_open($zipFile);
if (!is_resource($zip)) abort("File is not a valid ZIP format");

// Unpack the uploaded ZIP file into a tmpdir
$tempDir = tempnam(sys_get_temp_dir(), 'luac_dir');
if (file_exists($tempDir)) unlink($tempDir);
mkdir($tempDir);

while (is_resource($entry = zip_read($zip))) {
  $name = basename(zip_entry_name($entry));
  $entryName = "${tempDir}/${name}";

  // Filenames must have a valid Lua name root
  if (preg_match('/^[[:alpha:]_]\w*.lua$/', $name)) {
    echo "$entryName found\n"; // Debug
    $entries[] = $entryName;
    $contents = '';
    while (($chunk = zip_entry_read($entry,8192))) {
      $contents .= $chunk;
    }
    zip_entry_close($entry); $entry = NULL;
    file_put_contents($entryName, $contents);
  }
}
zip_close($zip); $zip = NULL;

if (count($entries)==0) abort("ZIP contains no Lua files");

print_r($entries); // Debug

// Now call luac.cross to do the compile
$luaFiles = implode (" ", $entries);
$outLFS = "${tempDir}/lfs.img";
exec("luac.cross -f -o ${outLFS} ${luaFiles}", $output, $status);

if ($status != 0) abort("luac.cross compile error: \n" . implode ("\n", $output) );

$LFSimage = file_get_contents($outLFS);
exec("rm -rf ${tempDir}");

file_put_contents($dbgErrLog, ob_get_contents());
ob_end_clean();

http_response_code(200);
header('Content-Description: File Transfer');
header('Content-Type: application/octet-stream');
header("Content-Disposition: attachment; filename=\"${imageName}\"");
header('Content-Transfer-Encoding: binary');
header('Expires: 0');
header('Cache-Control: must-revalidate, no-store, must-revalidate, max-age=0');
header('Pragma: public');
header('Content-Length: ' . strlen($LFSimage));
echo $LFSimage;

This Post Has 2 Comments

  1. Hey Terry, firstly I found your story fascinating and it’s great to see your mind very active and your skills being applied to the IOT!! Thanks. Secondly, your work on LFS is greatly appreciated and very timely … I’ve been battling the small ram footprint of the 8266 for a while and your solution is a godsend.

    However, I’m having trouble getting an LFS image over 64KB to work. I’ve built the images using docker and have set SPIFFS_FIXED_LOCATION to 0x100000. All is good with an image < 64KB but anything larger causes the board to panic and endlessly reboot.

    I've looked around online and haven't found any tips or a solution … can you suggest a way forward?

    Thanks in advance,

    Hugh

  2. Hi, I have just tested LFS, and it is really a good improvment.

    I am developping an application that left me with around 6k of remaining heap, with a lot of effor to avoid the “not enough memory” and after setting the LFS, now runs with more than 35k .

    Thanks a lot for the web service, that set you up with LFS in no time.

Leave a Reply

Close Menu