@Stefan-de-Lange @Martijn-Kooijman
Sorry for spamming you again, but the script is not reliable. I could not repeat reliable results and only one message was sent last night (I am sending every 15 min.). Certain hex values or a combination of them still seem to be interpreted as control characters. I will try to find those “forbidden” numbers. Sending control characters such as CR or ETX are not causing issues and are just treated as binary. I may need to resort to Base syntax, but I don’t like sending the double amount of data.

@Stefan-de-Lange @Martijn-Kooijman
Thanks for the help, appreciated!

@Stefan-de-Lange
I can confirm that it is working now. I managed to send the CR code 16 times as binary code and it arrives as 0D0D0D… HEX code.
//Solution for sharing
//sending binary data in binary extended syntax for SARA module
//measure (here a t and h float are converted into scaled unsigned 16 bit temperature and humidity values)…
uint16_t temperature_coded=round((t+40)/ 100 * 65535); //scale: -40C=0, 60C=65535
uint16_t humidity_coded = round(h / 101 * 65535); //scale: 0%=0,101%=65535
char payload[payload_size]; //define array of char
//write coded measurements into the char array
payload[0] = (byte) ((temperature_coded & 0xFF00) >> 8 );
payload[1] = (byte) ((temperature_coded & 0X00FF) );
payload[2] = (byte) ((humidity_coded & 0xFF00) >> 8 );
payload[3] = (byte) ((humidity_coded & 0X00FF) );
//payload[…etc… sendcommand=“AT+USOST=0,“172.27.131.100”,15683,”+String(payload_size);
cmd(sendcommand); //cmd: see previous post
cmd(payload);

@Stefan-de-Lange: Base syntax works fine, but it needs double the amount of bytes than the binary extended syntax. This is not ethical to use for narrow band IOT solutions. It is strange that I wait for the @ but it still does not do the binary syntax. Do I need to send an additional @ before sending the binary data? The manual states that I just need to send <length> bytes of data.

@Stefan-de-Lange

This is the code to send the USOST command which waits for the “@”:
cmd(“AT+USOST=0,“172.27.131.100”,15683,”+String(payload_size);

String cmd(String cmd) {
l("cmd: " + cmd);
SerialSARA.println(cmd);
int timeout = TIMEOUT;
while (!SerialSARA.available() & timeout > 0) {
timeout–;
delay(1);
}
if (timeout == 0 || timeout < 0) {
l(“Timed out”);
return “”;
}
String result = “”;
do {
result += char(SerialSARA.read());
delay(5);
} while(SerialSARA.available());
l("rsp: " + result);
return result;
}

@Stefan-de-Lange I do wait for “@” and code is correct, otherwise I would not be able to send “<<” as explained in my previous post.
The holy grail is how to send binary in such a way that SARA treats it as binary (the old firmware in the SARA in the current Arduino requires one to do so).

@Martijn-Kooijman I did a new tests where I use SerialSARA.println(“<<”); “<” is ascii code for 3c hex unsigned, so “<<” = 15420 decimal. This is nicely received at TMobile as “3c3c”.
(I get a nice confirmation of SARA with +USOST: … OK
But I do not want to send “<<” (because it can contain control characters with other codes) instead I want to send binary 15420 or as text “3c3c”. The old firmware wants it binary but so far I do not manage. I hope it is clear now.

@Stefan-de-Lange Yes, I am first sending the command as text, then after receiving the @ I switch to binary to send the raw data.
//So I first open a UDP channel:
Udp.beginPacket(“172.27.131.100”,15683);
//Then I send the USOST command using ascii printing
sendcommand=“AT+USOST=0,“172.27.131.100”,15683,”+String(payload_size);
SerialSARA.println(sendcommand);
//Subsequently I receive a “@” from the SARA module, then write binary data
Udp.write((uint8_t*)&payload, payload_size);//
Udp.endPacket();

@Martijn-Kooijman At T-Mobile I see 41542b… the first two bytes are a temperature and these bytes are received at “Allthingstalk”. 4154 hex unsigned is 16724 decimal and Allthingstalk converts this to a temperature using the following equation: T=coded/65535*100-40 and it show -14.48, so these steps are fine. The problem is that I have never sent 4154.

@Martijn-Kooijman Yes this number arrives at https://portal.iot.t-mobile.nl/projects/1544/devices a few seconds after I send it. I tried sending sending other values but it remains the same.
Only when I use SerialSARA.println(payloadtest); (this time with payloadtest as a char array I get the right values. However if one of these chars contain a control character the communication stops which is understandable for a print statement). Using a SerialSARA.write(…) in a loop byte for byte did not solve this (nothing ever received).

Update: I can send data but at the IOT portal always the same number arrives:
41542b55534f53543d2d312c22302e30
I use Udp.write((uint8_t*)&payload, payload_size) after receiving the “@”.

Additional info: code that also does not send the data:
for (i=0;i<payload_size;i++){
while (!SerialSARA.available()){}
SerialSARA.write(payload[i]);
}
//Also not working:
Udp.beginPacket(“172.27.131.100”,15683);
Udp.write((const char*)&payload, payload_size);
Udp.endPacket();

@Stefan-de-Lange You were a great help with the commands for communicating with T-Mobile. I meanwhile have a program that seemed to work. However, I recently discovered that the message I send after the “@” is not always arriving. Today I discovered why: if the bytes contain certain print control characters it modifies the data and only a few bytes arrive at T-Mobile (or nothing). My script uses this function to send serial data to the SARA module (and works fine for AT+ commands, but not for binary data:
/**

• Send AT command to u-blox module
  */
  String cmd(String cmd) {
  l("cmd: " + cmd);

SerialSARA.println(cmd);

int timeout = TIMEOUT;
while (!SerialSARA.available() & timeout > 0) {
timeout–;
delay(1);
}

if (timeout == 0 || timeout < 0) {
l(“Timed out”);
return “”;
}

String result = “”;

do {
result += char(SerialSARA.read());
delay(5);
} while(SerialSARA.available());

l("rsp: " + result);

return result;
}

As you can see it uses a “print” statement, but I think it should be a “write” statement. So I changed cmd(payload) that is started after receiving the “@” the script into:
for (i=0;i<payload_size;i++){
if (SerialSARA.available()){
SerialSARA.write(payload[i]);
}
}
But still nothing arrives at T-Mobile with certain bytes (that translate to “ETX”, “SOH”, “CR” “LF”… Help very much appreciated!

@Martijn-Kooijman I solved it by checking the return message after sending “AT+USOCR=17,7000”. Once it is contains “+USOCR: 0” in the returning string I know it is ok:
(it takes about 5 s after a reset)
added code:

String check_sara=cmd("AT+USOCR=17,7000");
while ((check_sara.indexOf("+USOCR: 0",1)<0)) {
  delay(500);
  check_sara=cmd("AT+USOCR=17,7000");
  Serial.println(check_sara);//just for checking
}

@Martijn-Kooijman @Stefan-de-Lange
Arduino sent me a message that they have close my “firmware update” ticket. Their recommendation is to use the serial passthrough method as outlined in: https://forum.arduino.cc/index.php?topic=628808.15

@Martijn-Kooijman I tried AT+CSGT but only ERROR as a return value (also with the serialpassthrough script). I can’t set it either: AT+CSGT=1,“test” for example.
So I have no clue how I should be able to test when it is fine to start.
I include the relevant code here, that my help understanding my power and “delay time” problems. I have no clue how I should do this modem greetings test to get rid of these excessive delays which consume power.

#include <MKRNB.h>
#include <ArduinoLowPower.h>
#define LOG true
#define l(x) if (LOG) Serial.println(x);
#define DELAY (15 * 1000) //default 15 s
#define TIMEOUT (5 * 1000) //default 5 s
unsigned long baud = 115200;
String response;
// connection state
boolean notConnected = true;

void setup() {
// enable the POW_ON pin SARA module
pinMode(SARA_PWR_ON, OUTPUT);
digitalWrite(SARA_PWR_ON, HIGH);

// reset the ublox module
pinMode(SARA_RESETN, OUTPUT);
digitalWrite(SARA_RESETN, HIGH);
delay(100);
digitalWrite(SARA_RESETN, LOW);
Serial.begin(baud);
SerialSARA.begin(baud);
}
Loop{
delay(16000); //wait for SARA module to get ready (>65mA)
//do some measurements
cmd(“AT+USOCR=17,7000”); //opens a port
delay(4000); //need to wait here also
cmd(“AT+USOST=0,“172.27.131.100”,15683,16” //it returns “@”
cmd(payload);
digitalWrite(SARA_PWR_ON, LOW);
LowPower.sleep(900000); //(10 mA with modem on, 1.2 mA with modem off)
digitalWrite(SARA_PWR_ON, HIGH);
}
/**

• Send AT command to u-blox module
  */
  String cmd(String cmd) {
  l("cmd: " + cmd);

SerialSARA.println(cmd);

int timeout = TIMEOUT;
while (!SerialSARA.available() & timeout > 0) {
timeout–;
delay(1);
}

if (timeout == 0 || timeout < 0) {
l(“Timed out”);
return “”;
}

String result = “”;

do {
result += char(SerialSARA.read());
delay(5);
} while(SerialSARA.available());

l("rsp: " + result);

return result;
}

@Martijn-Kooijman Thanks for the quick response. I have found the sending data error (cmd: see a previous post)!
My correct code now is:
sendcommand=“AT+USOST=0,“172.27.131.100”,15683,”+String(payload_size);
cmd(sendcommand); //
cmd(payload);
I had a Serial.println(“…”); statement before cmd(payload), this was not a problem with payload size 6 but did not work with a larger payload! It seems that the Ublox module wants the data immidiately after initiating the AT+USOST… command.

I will look into power savings later this week. A 2Ah battery now lasts about a month and includes some solar charging days. I have a solar panel facing west and capable of charging a powerbank at >200 mA, but it only manages to charge at around 100 mA at the MKR1500. Note that the Arduino stops sending data at around 3.4 V of the LiPo, so one cannot use the full capacity!
Settings (every 15 minutes): 16 s at about 65 mA (switching on modem after sleep and a 4 s wait after opening a channel) , then about 880 s sleeping at 1.1 mA. I don’t think I can make the 16 s shorter or the 4 s delay shorter, because in the past this has lead to loss of data or double messages. But I will check again with the improved data sending with the 16 bytes.

@Martijn-Kooijman Some updates, I tried your suggestion to add -DVERY_LOW_POWER to boards.txt but the result was that the Arduino IDE did not compile anymore. So I re-installed the IDE and it works again.
I again measured power consumption but now with another meter and in sleep mode it still consumes 1.2 mA. Will try your other suggestions later. First I need to solve something else:
I added a few more sensors and I could not send more than 6 bytes using my script. So tried your messages writing using Udp.write((uint8_t*)&msg, 16); This worked fine, I did receive 16 bytes. However, I never managed to receive the HEX values. For example sending a 16 bit value of 43869 should be sent as AB5D, however it arrives as 4154. Help much appreciated!

@Martijn-Kooijman
I do not use the library, I send the commands to the serial ports using the following script below. Crucial for me is to add a 35 s delay after a reset (otherwise I get double messages, or mising messages, or wrong data), additionally I need the 4 s delay after opening port (before sending the message itself). I haven’t checked current again, will arrange a different meter for that.

/**

• Send AT command to u-blox module
  */
  String cmd(String cmd) {
  l("cmd: " + cmd);

SerialSARA.println(cmd);

int timeout = TIMEOUT;
while (!SerialSARA.available() & timeout > 0) {
timeout–;
delay(1);
}

if (timeout == 0 || timeout < 0) {
l(“Timed out”);
return “”;
}

String result = “”;

do {
result += char(SerialSARA.read());
delay(5);
} while(SerialSARA.available());

l("rsp: " + result);

return result;
}

@Martijn-Kooijman Thanks for the advice! Good to know that current can be lower. I will check mine again. Other issues I had was missing messages, up to 20%! I seem to have solved it by adding a 4 s delay after opening a port (between opening port and sending message). I did that about 12 h ago and still not a single message lost!