Execute Python Code in QM Using python.net

[Davider]

I created a function, pycall2, which uses the python.net library to execute Python code in QM. I encountered two problems:

1.The print function in Python does not output anything.
2.When the parameters provided to the pycall2 function are incorrect, an exception occurs that causes the process to hang, and QM must be restarted.

Prerequisites for Running the pycall2 Function

1.Download the file from https://www.nuget.org/packages/pythonx86/3.12.7, change its extension to .zip, and copy all files from the tools folder inside the package to the QM installation directory.
2.Download the file from https://www.nuget.org/packages/pythonnet/3.0.5, change its extension to .zip, and copy the Python.Runtime.dll file from the package to the QM installation directory.
3.Set the PYTHONHOME and PYTHONPATH environment variables to the QM installation directory.
4.Use the following code for testing.

I have been using this function for some time, and it works well in most cases with minimal configuration required.
Thanks in advance for any suggestions and help.


Function pycall_T
 

Code:

Copy      Help

out

str pyCode=
;def func1(a1, a2, a3):
;;;;;return a1 * a2 + a3
;
;def func2(a1, a2):
;;;;;return a1 * a2
;
;def func3(a1, a2):
;;;;;return a1 + a2
;
;def noArgsFunc():
;,return 'no Args Func'
;
;def print2():
;,print('print output')

str f2="func2"; int a1(6) a2(7)
mes pycall2(pyCode, f2, a1, a2) ;;ok

mes pycall2(pyCode, "noArgsFunc") ;;ok

mes pycall2(pyCode, "print2") ;;no output

mes pycall2(pyCode, "8888888") ;;An error will occur, and the macro cannot be executed again unless qm is restarted.

Function pycall2

Code:

Copy      Help

function` str'pyCode [str'_Func] [`_A1] [`_A2] [`_A3] [`_A4] [`_A5]

CsScript x

str Opt=
;searchDirs=$qm$
;references=System.Core;Python.Runtime.dll;Microsoft.CSharp;netstandard
;compilerOptions=/warn:1
x.SetOptions(Opt)

x.AddCode("")

ARRAY(VARIANT) args.create
sel getopt(nargs)-2
,case 1 args[]=_A1
,case 2 args[]=_A1; args[]=_A2
,case 3 args[]=_A1; args[]=_A2; args[]=_A3
,case 4 args[]=_A1; args[]=_A2; args[]=_A3; args[]=_A4
,case 5 args[]=_A1; args[]=_A2; args[]=_A3; args[]=_A4; args[]=_A5

ret x.Call("py.pyCall" pyCode _Func args)

#ret
using System;
using System.IO;
using System.Diagnostics;
using System.Reflection;
using System.Linq;
using Python.Runtime;

public class py
{
,public static object pyCall(string code, string func = null, params object[] args) 
,{
,,dynamic Rd = pyRun(code, func, args); 
,,object R = Rd.ToString();
,,PythonEngine.Shutdown();
,,return R;
,}

,public static dynamic pyRun(string code, string func = null, params object[] args)
,{
,,PythonEngine.Initialize();
,
,,using (Py.GIL())
,,{
,,,dynamic result;
,,,if (string.IsNullOrEmpty(func) && (args == null || args.Length == 0))
,,,{
,,,,// Redirect output
,,,,dynamic sys = Py.Import("sys"); 
,,,,dynamic io = Py.Import("io");
,,,,var output = io.StringIO(); // Create a new StringIO object
,,,,sys.stdout = output; // Redirect sys.stdout to the StringIO object
,,,,
,,,,PythonEngine.RunSimpleString(code);
,,,,
,,,,result = output.getvalue(); // Get the value from StringIO
,,,,sys.stdout = sys.__stdout__; // Restore the original output
,,,}
,,,else
,,,{
,,,,using (var scope = Py.CreateScope())
,,,,{
,,,,,if (args != null && args.Length > 0)
,,,,,{
,,,,,,var pyArgs = new PyObject[args.Length];
,,,,,,for (int i = 0; i < args.Length; i++)
,,,,,,{
,,,,,,,pyArgs[i] = args[i].ToPython();
,,,,,,}
,,,,,,scope.Exec(code);
,,,,,,result = scope.Get(func).Invoke(pyArgs);
,,,,,}
,,,,,else
,,,,,{
,,,,,,scope.Exec(code);
,,,,,,result = scope.Get(func).Invoke();
,,,,,}
,,,,}
,,,}
,,,return result;
,,}
,,// PythonEngine.Shutdown();
,}
}

[Gintaras]

Maybe here can be used C# code from the LA Pynet class. It redirects print.
https://www.libreautomate.com/forum/show...5#pid36975

[Davider]

I previously tried having ChatGPT modify the code from that link and ran it in QM, but it was unsuccessful. The code above was also written by ChatGPT, but it is not perfect.

---

The code in #1 is a static function call, which is simple and straightforward to use.
The LA code uses instance function calls, which is a bit more complicated since you need to create an object first.
I rarely use instance function calls in qm. Can you modify the python.net code from LA to use instance calls in qm?

---

My ultimate goal is to be able to run Python code in qm like the example below—it’s just so simple to use.

Code:

Copy      Help

str pyCode= 
;def func2(a1, a2):
;;;;;return a1 * a2

str f2="func2"; int a1(6) a2(7)
mes pycall(pyCode, f2, a1, a2)

---

I found that running Python code in qm is much faster compared to running PowerShell code. Also, various AI programming assistants have the highest accuracy when writing Python code. That’s why I’m very persistent about implementing the approach used in the #1 code above.

[Davider]

Issue2 in #1 has already been resolved.

        try
        {
            dynamic Rd = pyRun(code, func, args);
            object R = Rd.ToString();
            PythonEngine.Shutdown();
            return R;
        }
        catch (Exception ex)
        {
            PythonEngine.Shutdown();
            //return "Error: " + ex.ToString();
            return null;
        }

[Davider]

How should the args array be assigned when there is no function name and no parameters?
_______________________________________________________________
ARRAY(VARIANT) args.create
sel getopt(nargs)-2
    case 0 args[]=?????
    case 1 args[]=_A1
    case 2 args[]=_A1; args[]=_A2
    case 3 args[]=_A1; args[]=_A2; args[]=_A3
    case 4 args[]=_A1; args[]=_A2; args[]=_A3; args[]=_A4
    case 5 args[]=_A1; args[]=_A2; args[]=_A3; args[]=_A4; args[]=_A5

ret x.Call("py.pyCall" pyCode _Func args)
_______________________________________________________________

Code:

Copy      Help

str pyCode=
;print('print output')
mes pycall2(pyCode) ;;noFunc noArgs

[Gintaras]

Function VariantArgsToArray

Code:

Copy      Help

;/
function ARRAY(VARIANT)&a VARIANT&firstParam nParamsBefore

;Creates ARRAY(VARIANT) a and moves VARIANT parameters of the caller function to it.

;a - ARRAY(VARIANT) variable. This function creates the array.
;firstParam - address of the first VARIANT parameter.
;nParamsBefore - the number of function parameters before the first VARIANT parameter.

;EXAMPLE
;function` $name [`_A1] [`_A2] [`_A3] [`_A4] [`_A5] [`_A6]
;ARRAY(VARIANT) a; VariantArgsToArray a &_A1 1

int na = getopt(nargs 1)-nParamsBefore
a.create(na)
if na>0
,na*sizeof(VARIANT)
,memcpy &a[0] &firstParam na
,memset &firstParam 0 na

[Davider]

When the function name and parameters are empty, it still shows an "invalid pointer" error.

https://i.ibb.co/fVCxxrjt/pycall.gif

[Gintaras]

Replace the a.create line with
a.create(iif(na<0 0 na))

Why invalid pointer error - I don't know. I can't test.

[Davider]

I added a static function named pyCall to the Pynet class. It works very well when testing the example code in LA, but it doesn't work in QM.

string code = """
def Multi(a1, a2):
    return a1 * a2
""";
var a = Pynet.pyCall(code, "Multi", 6, 7);
print.it(a);
 

Code:

Copy      Help

public static object pyCall(string code, string func = null, params object[] args)
{
    using (var pyn = new Pynet(code))
    {
        dynamic m = pyn.Module;
        var pyFunc = m.GetAttr(func);

        PyObject[] pyArgs = args.Select(arg => arg.ToPython()).ToArray();
        using (var result = pyFunc.Invoke(pyArgs))
        {
            return result.AsManagedObject(typeof(object));
        }
    }
}

---

qm code: 

Code:

Copy      Help

ARRAY(VARIANT) args.create
sel getopt(nargs)-1
,case 1 args[]=_A1
,case 2 args[]=_A1; args[]=_A2
,case 3 args[]=_A1; args[]=_A2; args[]=_A3
,case 4 args[]=_A1; args[]=_A2; args[]=_A3; args[]=_A4
,case 5 args[]=_A1; args[]=_A2; args[]=_A3; args[]=_A4; args[]=_A5
ret x.Call("Pynet.pyCall" code _Func args)

QM Err info:
0x80004002, Interface not supported.
The specified conversion is invalid.

[Davider]

thank you!

function name need set ""

[Gintaras]

QM Err info:
0x80004002, Interface not supported.
The specified conversion is invalid.

From C# to QM can be returned only simple types that can be stored in VARIANT, like string or int. Probably `result.AsManagedObject` returns some other type. Try in C# to get the type and convert to string etc.

[Davider]

The following code works, though I haven’t done extensive testing.
Can a list object be stored using a QM array?
How should JSON objects be handled? This is used frequently — is it possible to convert them into XML objects?
Handling a few common object types would be sufficient, such as: strings, integers, string arrays, and JSON strings.

Code:

Copy      Help

               switch (result.GetPythonType().Name)
                {
                    case "bool":
                        return result.As<bool>();
                    case "int":
                        return result.As<int>();
                    case "float":
                        return result.As<double>();
                    case "str":
                        return result.As<string>();
                    case "list":
                        var list = new List<object>();
                        dynamic pyList = result;
                        foreach (var item in pyList)
                        {
                            list.Add(item);
                        }
                        return list;
                    case "dict":
                        var dict = new Dictionary<object, object>();
                        dynamic pyDict = result;
                        foreach (var key in pyDict.keys())
                        {
                            dict[key] = pyDict[key];
                        }
                        return dict;
                    default:
                        try
                        {
                            return result.As<object>();
                        }
                        catch
                        {
                            return result.ToString();
                        }
                }

[Gintaras]

return list.ToArray();

But cannot return a dictionary. Try to convert to array that contains keys and values.

[Gintaras]

QM does not have a JSON class or a parser function. But in C# you can easily convert JSON to XML.
https://www.libreautomate.com/forum/show...8#pid26648

[Davider]

Running the same `Pynet` class code in QM and LA shows different behavior for the `print` statement:

1. In QM's output panel: each printed line is followed by two extra blank lines, as shown in the image.   https://i.ibb.co/G4qx7d6b/pyqm.png
2. In LA's output panel: there are no extra blank lines. https://i.ibb.co/35J1jqNH/prla.png

What is happening here?
 

Code:

Copy      Help

string code = """
###pywin32
import ctypes
print("hello world from python!")

import win32print
print(win32print.__file__)

default_printer = win32print.GetDefaultPrinter()
print(default_printer)
""";

using var pyn = new Pynet();
pyn.Module.Exec(code);

[Gintaras]

Please post full code for testing.

[Davider]

It may be necessary to upgrade csc.exe, as the system's default C# 5 cannot execute the following code.

demo:
https://i.ibb.co/chzkdkg3/pycall.gif

Function Pynet2

Code:

Copy      Help

out

CsScript x
str Opt=
;searchDirs=$qm$
;references=System.Core;Python.Runtime.dll;Microsoft.CSharp;netstandard
;compilerOptions=/warn:1
x.SetOptions(Opt 1)

x.AddCode("")
str code=
;###pywin32
;import ctypes
;print("hello world from python!")
;
;import win32print
;print(win32print.__file__)
;
;default_printer = win32print.GetDefaultPrinter()
;print(default_printer)

x.Call("Pynet.pyCall" code)


#ret
using System;
using System.IO;
using System.Diagnostics;
using System.Reflection;
using System.Linq;
using Python.Runtime;
using System.Collections;
using System.Collections.Generic;

public class Pynet : IDisposable
{
,Py.GILState _gil;
,PyModule _m;
,bool _shutdown;

,public static object pyCall(string code)
,{
,,,using (var pyn = new Pynet())
,,,{
,,,,pyn.Module.Exec(code);
,,,,return null;
,,,}
,}

,/// <summary>
,/// Initializes the Python engine and optionally adds code.
,/// </summary>
,/// <param name="code">If not null/"", calls <b>PyModule.Exec</b>.</param>
,/// <param name="enablePrint">Redirect the output of the Python's print function to console. Default true.</param>
,public Pynet(string code = null, bool enablePrint = true)
,{
,,if (!PythonEngine.IsInitialized)
,,{
,,,string pyDll = @"C:\Program Files (x86)\Quick Macros 2\python312.dll";
,,,if (string.IsNullOrEmpty(Environment.GetEnvironmentVariable("PYTHONNET_PYDLL"))) Runtime.PythonDLL = pyDll;

,,,PythonEngine.Initialize();
,,,_shutdown = true;
,,,//process.thisProcessExit += _ => PythonEngine.Shutdown(); //does not work
,,}
,,_gil = Py.GIL();

,,if (enablePrint)
,,{
,,,Module.Exec("""
import sys

class NetConsole(object):
;;;;def __init__(self, writeCallback):
;;;;;;;;self.writeCallback = writeCallback

;;;;def write(self, message):
;;;;;;;;self.writeCallback(message)

;;;;def flush(self):
;;;;;;;;pass

def setConsoleOut(writeCallback):
;;;;sys.stdout = NetConsole(writeCallback)
""");
,,,var writer = (string s) => { Console.Write(s); };
,,,Module.InvokeMethod("setConsoleOut", Python.Runtime.PyObject.FromManagedObject(writer));
,,}

,,if (!string.IsNullOrEmpty(code)) Module.Exec(code);
,}

,/// <summary>
,/// Unlocks/disposes Python objects and shuts down the engine.
,/// </summary>
,public void Dispose()
,{
,,_m?.Dispose();
,,_gil?.Dispose();
,,if (_shutdown)
,,{
,,,try { PythonEngine.Shutdown(); } //without this the process does not exit
,,,catch (System.Runtime.Serialization.SerializationException) { } //thrown when using enablePrint
,,,//.NET 9+: BinaryFormatter serialization and deserialization have been removed
,,,//catch (PlatformNotSupportedException) { }
,,}
,}

,/// <summary>
,/// Gets the result of <b>Py.CreateScope</b>.
,/// You can assign it to a <c>dynamic</c> variable and call functions defined in your Python code.
,/// </summary>
,public PyModule Module => _m ??= Py.CreateScope();
}

[Gintaras]

Converted to C# 6. Fixed newlines.
But with `import win32print` throws error `PythonEngine is not initialized`. Don't know why.
Note, I changed the searchDirs path and the dll path.

Macro Macro3510

Code:

Copy      Help

/exe
out

CsScript x
str Opt=
;searchDirs=C:\LA workspace\.nuget\PyNet
;references=System.Core;Python.Runtime.dll;Microsoft.CSharp;netstandard
;compilerOptions=/warn:1
x.SetOptions(Opt 1)

x.AddCode("")
str code=
;###pywin32
;import ctypes
;print("hello world from python!")
;print("goodbye!")

x.Call("Pynet.pyCall" code)
;
;import win32print
;print(win32print.__file__)
;
;default_printer = win32print.GetDefaultPrinter()
;print(default_printer)


;BEGIN PROJECT
;main_function  Macro3510
;exe_file  $my qm$\Macro3510.qmm
;flags  6
;guid  {E66B3D29-E971-42A1-A722-B304704DB56D}
;END PROJECT

#ret
using System;
using System.IO;
using System.Diagnostics;
using System.Reflection;
using System.Linq;
using Python.Runtime;
using System.Collections;
using System.Collections.Generic;

public class Pynet : IDisposable
{
,Py.GILState _gil;
,PyModule _m;
,bool _shutdown;

,public static object pyCall(string code)
,{
,,,using (var pyn = new Pynet())
,,,{
,,,,pyn.Module.Exec(code);
,,,,return null;
,,,}
,}

,/// <summary>
,/// Initializes the Python engine and optionally adds code.
,/// </summary>
,/// <param name="code">If not null/"", calls <b>PyModule.Exec</b>.</param>
,/// <param name="enablePrint">Redirect the output of the Python's print function to console. Default true.</param>
,public Pynet(string code = null, bool enablePrint = true)
,{
,,if (!PythonEngine.IsInitialized)
,,{
,,,Runtime.PythonDLL = @"C:\Program Files (x86)\Python313-32\python313.dll";

,,,PythonEngine.Initialize();
,,,_shutdown = true;
,,}
,,_gil = Py.GIL();

,,if (enablePrint)
,,{
,,,Module.Exec("import sys\nclass NetConsole(object):\n    def __init__(self, writeCallback):\n        self.writeCallback = writeCallback\n\n    def write(self, message):\n        self.writeCallback(message)\n\n    def flush(self):\n        pass\n\ndef setConsoleOut(writeCallback):\n    sys.stdout = NetConsole(writeCallback)\n");
,,,Action<string> writer = (string s) => { if(!(s=="\r" || s=="\n")) Console.Write(s); };
,,,Module.InvokeMethod("setConsoleOut", Python.Runtime.PyObject.FromManagedObject(writer));
,,}

,,if (!string.IsNullOrEmpty(code)) Module.Exec(code);
,}

,/// <summary>
,/// Unlocks/disposes Python objects and shuts down the engine.
,/// </summary>
,public void Dispose()
,{
,,if(_m!=null) _m.Dispose();
,,if(_gil!=null) _gil.Dispose();
,,if (_shutdown)
,,{
,,,try { PythonEngine.Shutdown(); } //without this the process does not exit
,,,catch (System.Runtime.Serialization.SerializationException) { } //thrown when using enablePrint
,,,//.NET 9+: BinaryFormatter serialization and deserialization have been removed
,,,//catch (PlatformNotSupportedException) { }
,,}
,}

,/// <summary>
,/// Gets the result of <b>Py.CreateScope</b>.
,/// You can assign it to a <c>dynamic</c> variable and call functions defined in your Python code.
,/// </summary>
,public PyModule Module { get { if(_m==null) _m = Py.CreateScope(); return _m; } }
}

[Davider]

it worked,  thank you so much!

[Gintaras]

JSON parsing:
https://www.libreautomate.com/forum/show...p?tid=7832