Asynchronous execution for Google App Scripts (gas)

Async.gs

/*
* Async.gs
*
* Manages asyncronous execution via time-based triggers.
*
* Note that execution normally takes 30-60s due to scheduling of the trigger.
*
* @see https://developers.google.com/apps-script/reference/script/clock-trigger-builder.html
*/

var Async = Async || {};
var GLOBAL = this;

// Triggers asynchronous execution of a function (with arguments as extra parameters)
Async.call = function(handlerName) {
  return Async.apply(handlerName, Array.prototype.slice.call(arguments, 1));
};

// Triggers asynchronous execution of a function (with arguments as an array)
Async.apply = function(handlerName, args) {
  var trigger = ScriptApp
  .newTrigger('Async_handler')
  .timeBased()
  .after(1)
  .create();
  CacheService.getScriptCache().put(String(trigger.getUniqueId()), JSON.stringify({ handlerName: handlerName, args: args }));
  return { 
    triggerUid: trigger.getUniqueId(),
    source: String(trigger.getTriggerSource()), 
    eventType: String(trigger.getEventType()), 
    handlerName: handlerName,
    args: args
  };
};

// GENERIC HANDLING BELOW
// 
function Async_handler(e) {
  var triggerUid = e && e.triggerUid;
  var cache = CacheService.getScriptCache().get(triggerUid);
  if (cache) {
    try {
      var event = JSON.parse(cache);
      var handlerName = event && event.handlerName;
      var args = event && event.args;
      if (handlerName) {
        var context, fn = handlerName.split('.').reduce(function(parent, prop) {
          context = parent;
          return parent && parent[prop];
        }, GLOBAL);
        if (!fn || !fn.apply) throw "Handler `" + handlerName + "` does not exist! Exiting..";
        // Execute with arguments
        fn.apply(context, args || []);
      }
    } catch (e) {
      console.error(e);
    }
  }
  // Delete the trigger, it only needs to be executed once
  ScriptApp.getProjectTriggers().forEach(function(t) {
    if (t.getUniqueId() === triggerUid) {
      ScriptApp.deleteTrigger(t);
    }
  });
};

Hi @Zained @edmundmaruhn, a bit late to the party... my 2c:

I would go with 4 (Store Files in Google Drive). Then just pass the file ID to your async function.

var file = DriveApp.getRootFolder().createFile(blob);
Async.call('processPDF', file.getId());

function processPDF(fileId) {
   // some logic
}

Alternatively, if the creating the blob or saving the file is the time-consuming task you want to avoid, then pass what you need in order to generate the PDF into an async task: ie

var data = { user: 1842559, order: 6621399 };
Async.call('generateReport', data);

function generateReport(data) {
   // some logic
}

Juste want to say that this is exactly what I was trying to do all day. I wouldn't have been able to do something like this! Thank you sooo much

Very cool! Please make the trigger name from the name of the function being executed. This way you can find the logs faster. I tried to specify a different trigger name, an error occurs.

Async.apply = function(handlerName, args) {
var trigger = ScriptApp
.newTrigger('Async_handler') // i tried 'String(args[0])'+'_Async_handler'

Thank you so much for this.

One more issue, my Function after that async update have a ping about 1-2 min. How can i minimize this time?

Hi @steevab:

See the comments:

* Note that execution normally takes 30-60s due to scheduling of the trigger.

There is just no way around that.

Hi @sdesalas,
First of all thanks, what you provide is really great, it's been a while since I try to find this kind of trick for scripts that are very long to run that I want to split into smaller batches and have them run asynchronously.
I just tried to apply it to one of these scipts and unfortunatly I don't get the expected result.
I was wondering if there were known limitations to have it work?

My script is quite simple :

function createBatchScript(num = 50){
  var total = getTotalNumber(); // Get the number of rows in a specific sheet with the data to treat
  var treated = getNumberTreated(); // Get the number of rows in another sheet with the result of the function
  var iterations = ((total - treated)/num).toFixed(0) + 1;
  for (var i = 0; i < iterations; i++){
    var start = treated + i * num + 1;
    Async.call(identifyRedirections,[start, num]); // Program the Trigger of the function for each batch
  }
}

I see the triggers properly created, I see the Async_handler functions running in the execution pannel, but the result of my function is not added to the result sheet as expected.

Do you have an idea what could happen or what I could do to debug?

Thanks in advance for your help

Hi @jberdah,

Triggers are expensive things for GAS to keep track off so I would avoid putting Async.call(blah) inside a loop.

A better way to acomplish this would be to put the for..loop inside the asynchronus function you are calling from within your createBatchScript function, so you dont create a zillion triggers inside a loop but just a single one.

Regards,

Is there a way to pass complex objects as arguments to Async.call ? My script works fine as long as I pass strings as parameters, but if I pass an object of type Spreadsheet, for example, the handler function sees an empty object instead of the Spreadsheet...

Hi @Tamarindo94

Only primitives will work as callback arguments, however its relatively straightforward to pass the ID and then recreate the Spreadsheet object inside the handler method.

Exception: This script has too many triggers. Triggers must be deleted from the script before more can be added.

I just copied your code
Did the error message appear because the google script was updated? Or am I doing something wrong?
Thanks

Hi @designer-1407

The most likely explanation for your script having too many triggers is because you are calling the Async function inside a loop.

See comment above

Triggers are expensive things for GAS to keep track off so I would avoid putting Async.call(blah) inside a loop.

A better way to acomplish this would be to put the for..loop inside the asynchronus function you are calling from within your createBatchScript function, so you dont create a zillion triggers inside a loop but just a single one.

You should be able to go into the list of Triggers for your script (side menu) and delete them one by one.

u should be able to go into the list of Triggers for your script (side menu) and del

Thank you so much for your reply!
I was looking for a solution to speed up data processing in my table by running several algorithms at once.
The solution was 2 things:

1. Run Google apps script in 2 browser tabs and run code in them from one file (it works even with 3 or more browser tabs) Such multi-threading =)
2. I switched to python, though. Everything is much faster there, and there is no limit on the algorithm in 6 minutes.

I tried implementing this code. If I call Async.apply directly while testing, it works fine. If placed then call into MyOnOpen.gs which is itself invoked by a trigger. This does not seem to work and I get the error message: 'Script function not found: Async_handler'
Can anybody provide me direction on how to fix this?

@sdesalas Hi! I tried to apply your async solution, but on first iteration an error pops up when calling the object's method. What do you think is the problem?

Error: [TypeError: folderSecond.createFile is not a function]

Code:
function uploadFilesFromDrive(url,filename,folderSecond){
const fileID = getIdFrom(url);
const getFile = DriveApp.getFileById(fileID);
const blob = getFile.getBlob().setName(filename).getAs('application/pdf');
const file = folderSecond.createFile(blob);
};

@ShadrinSpock see comment above

Only primitives will work as callback arguments, however its relatively straightforward to pass the ID and then recreate the Spreadsheet object inside the handler method.

https://gist.github.com/sdesalas/2972f8647897d5481fd8e01f03122805?permalink_comment_id=4428924#gistcomment-4428924

In your case you probably want the ID of the folder. Instead of the folder object.

@sdesalas Thank you so much for writing this! I've only just begun my journey as a developer and learned how to write asynchronous functions in Javascript today, only to find that it doesn't work in Google Apps Script. Your library worked like a charm for my purposes though. I managed to get the expected result on the first try, which is honestly incredibly surprising knowing me.

It seems so weird that you can write async function blah(), or return promises and write .then(), but Apps Script still runs it synchronously. Do you know why that is? What is the point of having the syntax if it doesn't work the way it was intended?

@mojoro https://stackoverflow.com/questions/31241396/is-google-apps-script-synchronous

Hi @sdesalas ,

We are trying to handle (to import data via Plaid API), large amount of data on the spreadsheet (google sheet).
The external API needs to be used in the while loop because response needs to be check the condition is true or false.
If the conditions meets true then the loop will continue until condition false.
And also the external API has limit restrictions(50 calls per minute).
We need to store all the response in a single variable like array collection then we need to format and manipulate them in the spreadsheets.

We have shared the code here for you.

//once trigger first time.
Async.call('updateSpreadSheet');

var responseData = [];

function updateSpreadSheet(){
  var collection = [];
  let response = fetchAPITransactions();
  if( response == true ){
   collection = fetchMoreAPITransactions();
  }
  if(collection.length > 0 ){
   manipulateDatatToSpreadsheet(collection);
  }
}

function manipulateDatatToSpreadsheet(){
 //format data and add/remove data on the spreadsheets.
}

function fetchMoreAPITransactions( response ){
 while( response == true ){
    responseData.push( response);
  break;
  }
  return responseData;
}
 
function fetchAPITransactions(){
 //call api end point and response. 50 calls per minute. 
 var response = responsedataofAPI;
 return response;
}

Is this approach correct for calling the Async function to execute the loop, the triggers are not called sequentially, this makes the data in the Spreadsheet also not in correct format. or not in sequence as per date order.
This process also runs for a very long time 45 to 60 minutes which is not practically feasible while retrieving the data.
What is the best approach in this case? We expect the data to be ordered by date in the spreadsheet and to fetch the data much quicker.

Hi @stallioninet,

I have not tested your approach, but I believe you don't need this library to achieve your desired result.

Google Apps Script has its own built-in API for making requests to external API's, and it will not execute code past the request if you are using their tools. With GAS, you don't need the traditional javascript async syntax to achieve some async results within standard functions. For example, every call you make to SpreadsheetApp or something similar is asynchronous, but the syntax doesn't require it. That is all handled under the hood.

Async.gs saves the function and the parameters you pass in at the time you call it and programmatically creates a time-based trigger that executes it as soon as possible. This is useful in many cases, such as when you need to execute things out of sequence, but you should not need it here. You would likely be better served by paginating through the data served by the API you are trying to contact and either writing it to the sheet as you receive it or storing it all in a 2-D array that is written after the pagination is complete.

Ensure that you use the built-in methods for interacting with external api's, run your code sequentially, and all should be well.