How to get the last message in Google Chrome browser

[Davider]

I want to get the last message in ChatGPT chat in Google Chrome browser as shown in the screenshot below. It's nested deep. I tried many methods but no luck.
https://i.ibb.co/r2LbX7g/66.png

In Qm, similar methods can be used. How to achieve this goal in LA?

Macro Macro31

Code:

Copy      Help

w=wait(6 WV win("xxx" "xxx"))
a.Find(w2 "LIST" "xxx" "" 0x1001 6)
ARRAY(Acc) ac
a.GetChildObjects(ac) 
Acc& r=ac[(ac.len)-1] ;
_s = r.Name

[Davider]

When the reply text consists of multiple lines, the nested levels become more complex
https://i.ibb.co/k4yB0zy/aaa.png

[Gintaras]

Code:

Copy      Help

// script ""
print.clear();

var w = wnd.find(1, "ChatGPT - Google Chrome", "Chrome_WidgetWin_1");
var e = w.Elm["web:GROUPING", prop: "@data-message-author-role=assistant", flags: EFFlags.Reverse].Find(1);

var s = e.GetInnerText();
print.it("---");
print.it(s);


static class Ext {
    /// <summary>
    /// Not perfect!
    /// </summary>
    public static string GetInnerText(this elm t) {
        var ae = t.Elm[null].FindAll();
        if (ae.Length == 0) return "";
        //print.it(ae);
        
        List<(string s, RECT r)> a = new();
        foreach (var e in ae) {
            if (e.GetProperties("nvr", out var p)) {
                var s = p.Name; if (s.NE()) s = p.Value;
                if (s.NE() || p.Rect.NoArea) continue;
                //print.it(e);
                if (a.Count > 0 && s == a[^1].s && a[^1].r.Contains(p.Rect)) a.RemoveAt(a.Count - 1);
                //if (p.Role == "GROUPING") continue;
                a.Add((s, p.Rect));
            }
        }
        
        var b = new StringBuilder();
        int i = -1, y = 0;
        foreach (var (s, r) in a) {
            i++;
            //print.it(s, r);
            if (b.Length > 0) if (r.top > y) b.AppendLine(); else b.Append(" ");
            y = r.bottom;
            b.Append(s);
        }
        
        return b.ToString();
    }
}

[Gintaras]

Or click the Copy button. It gets markdown text.

Code:

Copy      Help

// script ""
print.clear();

var w = wnd.find(1, "ChatGPT - Google Chrome", "Chrome_WidgetWin_1");
var e = w.Elm["web:GROUPING", prop: "@data-message-author-role=assistant", flags: EFFlags.Reverse, navig: "ne2 fi"].Find(1); //the navig finds the Copy button
clipboard.clear();

//e.Invoke(); //does not work
if (e.IsOffscreen) {
    e.ScrollTo();
    wait.until(3, () => !e.IsOffscreen);
    500.ms();
}
e.MouseClick();

var s = wait.until(3, () => clipboard.text);
print.it(s);

[Davider]

thank you so much!
#3 work,
#4 not work
https://i.ibb.co/nrj18fB/1.gif

Is there a way to make the code work in a headless browser, for example (e.MouseClick())

[Gintaras]

In my ChatGPT there are 4 buttons instead of "2/2" + 3 buttons.
Try different navig, for example
pa la pr2 fi

-----

If you have some code that uses a headless browser, please post.

[Gintaras]

Or you can use OpenAI API instead of a web browser. Like in the cookbook translator.

[Davider]

e.MouseClick(); //copy button

var s = wait.until(3, () => clipboard.text); //Error

The line of code has a certain probability of encountering errors during actual execution, which is random.

[Gintaras]

Maybe your script is too fast, and the Copy button is still not there or still does not work. Insert some delays.

[Davider]

Method #3 isn't always perfect.
I just noticed the Playwright automation library. Would using it be more stable?
Additionally, browser automation seems to have become mainstream in the automation world. Will LA add a browser recorder feature in the future? Or are there any similar, recommended libraries available?

[Gintaras]

With Selenium text extraction results are much better. It's in Cookbook. I didn't test extracting text with Playwright.

Reliable recording is impossible. Something works, something not. LA will not have a web recorder soon.

Selenium has a web recorder Selenium IDE. It can export C# code that can be used in LA. Something similar exists for Playwright, but I didn't test.

[Gintaras]

How to use Selenium IDE web recorder results in LA.

Selenium IDE records a Test. Right-click the test and export as C# xUnit.
Import it into LA. For example drag drop.
In Properties change file type from class to script.

The code is like:
 

Code:

Copy      Help

// script "T1Test.cs"
// Generated by Selenium IDE
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading;
using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;
using OpenQA.Selenium.Firefox;
using OpenQA.Selenium.Remote;
using OpenQA.Selenium.Support.UI;
using OpenQA.Selenium.Interactions;
using Xunit;
public class SuiteTests : IDisposable {
  public IWebDriver driver {get; private set;}
  public IDictionary<String, Object> vars {get; private set;}
  public IJavaScriptExecutor js {get; private set;}
  public SuiteTests()
  {
    driver = new ChromeDriver();
    js = (IJavaScriptExecutor)driver;
    vars = new Dictionary<String, Object>();
  }
  public void Dispose()
  {
    driver.Quit();
  }
  [Fact]
  public void T1() {
    driver.Navigate().GoToUrl("https://www.libreautomate.com/");
    driver.Manage().Window.Size = new System.Drawing.Size(1104, 723);
    driver.FindElement(By.LinkText("Forum")).Click();
    driver.FindElement(By.LinkText("Ideas (LA)")).Click();
    driver.FindElement(By.LinkText("LA Context Menu")).Click();
  }
}

Edit:
 

Code:

Copy      Help

// script "T1Test.cs"
/*/ nuget selenium\Selenium.Support; nuget selenium\Selenium.WebDriver.ChromeDriver; /*/
// Generated by Selenium IDE
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading;
using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;
using OpenQA.Selenium.Firefox;
using OpenQA.Selenium.Remote;
using OpenQA.Selenium.Support.UI;
using OpenQA.Selenium.Interactions;

using var t = new SuiteTests();
t.T1();

public class SuiteTests : IDisposable {
    public IWebDriver driver { get; private set; }
    public IDictionary<String, Object> vars { get; private set; }
    public IJavaScriptExecutor js { get; private set; }
    public SuiteTests() {
        var service = ChromeDriverService.CreateDefaultService(pathname.getDirectory(typeof(ChromeDriver).Assembly.Location));
        service.HideCommandPromptWindow = true;
        driver = new ChromeDriver(service);
        js = (IJavaScriptExecutor)driver;
        vars = new Dictionary<String, Object>();
    }
    public void Dispose() {
        driver.Quit();
    }
    
    public void T1() {
        driver.Navigate().GoToUrl("https://www.libreautomate.com/");
        driver.Manage().Window.Size = new System.Drawing.Size(1104, 723);
        driver.FindElement(By.LinkText("Forum")).Click();
        driver.FindElement(By.LinkText("Ideas (LA)")).Click();
        driver.FindElement(By.LinkText("LA Context Menu")).Click();
    }
}

If does not work, install or update the 2 NuGet packages. Need to update the NuGet packages for each major Chrome version.
I install Selenium packages in folder "selenium". Edit the first line if installed elsewhere.

[Davider]

Thank you for sharing!
It's a bit complicated, and I didn't succeed in testing it.
PuppeteerSharp and Playwright seem easier to use. I haven't found recording tools for these two libraries yet.
If possible, LA's web recorder generating Playwright code would be better.

---

I found a video demonstration of the Playwright recorder in VSC, and it looks really great.
https://youtu.be/LM4yqrOzmFE

[Davider]

#4 Code

e.MouseClick();

Below error frequently occurs, a delay is added before this line of code( 500.ms();).

Au.Types.AuException: The UI element rectangle is not in the container window. Try scroll.

[Davider]

During extensive testing, this issue occurs very frequently. Similarly, using 'goto' labels can solve it,
but the code would become cluttered with too many 'goto' statements.

[Gintaras]

Difficult to find the button element reliably with elm. No name etc; the UI element tree is different in different ChatGPT states. Even key End does not work.

This script uses another way. At first scrolls to the first element below text (the Copy button probably is in that row). Then uses uiimage.
 

Code:

Copy      Help

print.clear();

var w = wnd.find(0, "*Google Chrome", "Chrome_WidgetWin_1");

//scroll to the first element after the text
var e = w.Elm["web:GROUPING", prop: "@data-message-author-role=assistant", navig: "ne"].Find(1);
e.ScrollTo();

clipboard.clear();

string image = @"image:";
var im = uiimage.find(3, w, image, IFFlags.PrintWindow);
im.MouseClick();
mouse.moveBy(0, -50);

var s = wait.until(3, () => clipboard.text);
print.it(s);

Recapture the Copy button image.

[Davider]

Thanks for your help.
typically use image scanning infrequently to capture controls, which can sometimes lead to errors, such as varying resolutions, using remote desktop, system or chatgpt theme changes, etc.