cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
BrendonKoz
Advocate I
Advocate I

Error with JSON string as passed from PAD to "Run Python Script"

I'm trying to take advantage of an external script's ability to perform list filtering faster than a multi-iteration loop within PAD. I have similar objects that I am comparing, each with an id property. If the "handled" list contains a matching ID to the "events" list, I want to remove that particular entry in the object. I've gotten this working in both plain JavaScript, as well as Python, but no matter what I do, I keep getting errors. The JavaScript compiler's error messages were too vague, so I switched to Python. In Python, I seem to be getting an error where the original JSON string's values are being stripped of its surrounding double quotes. This is causing Python interpreter errors. Am I handling this wrong?

 

Example of working Python, outside of PAD, (with most extraneous values removed for brevity):

events = [{"id":"11438923","title":"Event One"},{"id":"11362898","title":"Event Two"},{"id":"10734368","title":"Event Three"}]
handled = [{'id':'11438923'}]

# Extract the ids from the second object
handled_ids = set([obj['id'] for obj in handled])

# Filter out duplicates from the first object
events = [e for e in events if e['id'] not in handled_ids]

print(events)

The resulting list from the above code simply removes the first entry (matching the ID) from the events variable, and prints it out.

 

Although slightly adjusted from the above code, to handle how the values exist within my PAD flows, once run in PAD, the double-quoted strings for the object's values are seemingly gone. I can't even get past the assignment of the variable before an error is thrown. Immediately prior to entering the "Run Python script" action, if I display a Message Box with the content of the variable, the double-quotes exist.

 

Example input (JsonEvents variable, limited to one imported entry just to show its content) immediately prior to the Python action:

{"events":[{"id":"11438923","allday":"1","multiday":"1","private":"0","start":"1701406800","end":"1703393999","setup":"0","breakdown":"0","title":"Festive Fun with Felt: December Take-and-Make Craft Kits","description":"<p>Enjoy a seasonal take-and-make felt kit for December! Celebrate the start of winter and select one of three project options: Red Cardinal, Gingerbread House, or Knotted Tree. Each kit provides the pattern, directions, and materials to create and enjoy a festive felt craft. <\/p>\r\n\r\n<p>Pick up a free craft kit at the Library&rsquo;s Information Desk any day this month! One kit per visit, please; available while supplies last.<\/p>","more_info":"<p>Free printable <a href=\"https:\/\/guides.sspl.org\/FeltFun\">directions and patterns for &ldquo;Take and Make&rdquo; kits<\/a> are always available online for current and past projects.<\/p>","url":"https:\/\/sspl.libcal.com\/event\/11438923","admin_url":"https:\/\/sspl.libcal.com\/aevent.php?c=10655&m=edit&event_id=11438923","location_id":"0","location":"","audience_id":"1604|1623|1627|1628","audience":"Children|Adults|Teens|All Ages","cat_id":"64132","category":"Art, Music, and Movies > Crafting","owner":"Kali Nagler","presenter":"Miriam Meier & Jasmine Gage","cal_id":"10655","calendar":"Library Events","color":"#B3C8EF","image":"https:\/\/lcimages.s3.amazonaws.com\/data\/feat_img\/4700\/10655\/1697730913.png","geo_id":"ChIJh9RRqdA53okR1XeeOfmgCng","geo_lat":null,"geo_long":null,"cost":"0","registration":"0","registration_open":"0","seats":"","seats_taken":null,"physical_seats":null,"physical_seats_taken":null,"online_seats":null,"online_seats_taken":null,"zoom_email":null,"online_user_id":null,"online_meeting_id":null,"online_host_url":null,"online_join_url":null,"online_join_password":null,"online_provider":null,"wait_list":"0","future_dates":null}]}

 

The Python interpreter error I get when I run the action:

  File "", line 3
    handled = { 'id': 11438923, 'allday': 1, 'multiday': 1, 'private': 0, 'start': 1701406800, 'end': 1703393999, 'setup': 0, 'breakdown': 0, 'title': Festive Fun with Felt: December Take-and-Make Craft Kits, 'description': <p>Enjoy a seasonal take-and-make felt kit for December! Celebrate the start of winter and select one of three project options: Red Cardinal, Gingerbread House, or Knotted Tree. Each kit provides the pattern, directions, and materials to create and enjoy a festive felt craft. </p>
                                                                                                                                                               ^
SyntaxError: unexpected token 'Fun'

 

Ignoring the second variable for now, here are the two actions in the flow to cause the issue:

BrendonKoz_0-1701902807008.png

...and the content of the Python script:

events = %JsonEvents%
events = events['events']
handled = %HandledEventsTable%

# Extract the ids from the second object
handled_ids = set([h['id'] for h in handled])

# Filter out duplicates from the first object
events = [e for e in events if e['id'] not in handled_ids]

print(events)

 

Are there any thoughts as to what I might be missing here?

1 ACCEPTED SOLUTION

Accepted Solutions
BrendonKoz
Advocate I
Advocate I

In the end, I'm not 100% sure what worked to get around my initial issue; it never appeared again. @Agnius' tip about how to reference Python modules was exactly what I needed to know to properly import modules (I wrongly assumed since PAD supposedly used IronPython, that it was already preinstalled somewhere.) I was unable to get PAD's Python to properly convert the JSON string into a Python object; it might've been that my formatting of the string was wrong, or there was something else that I was missing...either way, I found a trick to handle nulls that simply declared `null` as a variable, and assigned it Python's null value (`null = None`) which solved my null value issue in converting the JSON to a Python object (they're so similar, it just became an object, essentially).

 

 

 

 

import json

null = None

events = %JsonEvents%
handled = %HandledEventsTable%
events = events['events']

# Extract the ids from the second object
handled_ids = set([h['id'] for h in handled])

# Filter out duplicates from the first object
events = [e for e in events if e['id'] not in handled_ids]

print(json.dumps(events))

 

 

 

 

After exporting the string back to PAD via the `print` statement, I noticed there was a BOM character preceeding the intended value. In order to strip this, I used a simple find/replace REGEX, primarily because I couldn't figure out how to prevent the BOM from being returned from within the Python code itself.

 

Text.Replace Text: PythonOutput TextToFind: $'''^.*\\[''' IsRegEx: True IgnoreCase: True ReplaceWith: $'''[''' ActivateEscapeSequences: True Result=> PythonOutput

 

NOTE: PythonOutput is the variable name that the "Run Python script" action created in this example.

 

After that, I converted the JSON string back to a custom object for PAD to continue doing what it needed to do with my list of custom objects.

 

Definitely a hacky solution that I'm not 100% proud of, but sometimes we just have to move forward and don't have time to look back. I still don't know why I was having string quote issues with my first attempts, and since that was never really solved, I'm not sure if marking any response here as a solution is appropriate, but I wanted to follow up and at least provide what my solution ended up being in case it helps someone else in the future.

 

Additional notes:

  1. PAD doesn't (currently) handle null values as provided initially from a JSON string all too well. If accessing a property that has a null value, PAD will error and say the property doesn't exist (in fact its value doesn't exist). To workaround that, change the value of `null` to an empty string.
    BrendonKoz_0-1702407662482.png
  2. Python, as how my code is written above, will double-escape backslashes, which were inserted to escape forward slashes. During conversion back, this causes the string to remain escaped (ex: http:// would instead be http:\/\/), so before converting from a JSON string back to a custom object in PAD, I had to replace "\\" with "\" (non-regex).
  3. Be aware with any/all string replacements that contains external content; you might unintentionally replace something you didn't intend to. The word part of "null" might've actually been intended within usable textual content. In my case that is unlikely, but your case might be different. If that's the scenario, your find/replace might need to be different in order to account for that.

View solution in original post

9 REPLIES 9
lbendlin
Multi Super User
Multi Super User

Strings need to be enclosed in quotes.  Your 'title'  value and others are not.  Depending on the implementation you can use single or double quotes, but you must choose one or the other.

That's the thing though - they are enclosed in double quotes prior to being passed into the Python action (as are the property names, such as 'id', which retain their quotes for some reason). The 2nd code block above is an example of the text from the variable as output from a message box, immediately prior to running the Python script action.

Agnius
Most Valuable Professional
Most Valuable Professional

What's in the "%HandledEventsTable%" variable? It seems like that's a data table and not just a JSON blob. That will obviously not work, as a data table in PAD will not be interpreted as a data table in Python when you pass it in. You need to pass in a single string that can then be processed by Python. So, possibly best to pass it in as a JSON string as well. 

-------------------------------------------------------------------------
If I have answered your question, please mark it as the preferred solution. If you like my response, please give it a Thumbs Up.

I also provide paid consultancy and development services using Power Automate. If you're interested, DM me and we can discuss it.

-------------------------------------------------------------------------------------------------------------------------
If I have answered your question, please mark it as the preferred solution. If you like my response, please give it a Thumbs Up.
Regards, Agnius Bartninkas

The value of %HandledEventsTable% at the time of instantiation (I reused the variable and converted it):

[{
  "id": "11438923",
  "allday": "1",
  "multiday": "1",
  "private": "0",
  "start": "1701406800",
  "end": "1703393999",
  "setup": "0",
  "breakdown": "0",
  "title": "Festive Fun with Felt: December Take-and-Make Craft Kits",
  "description": "<p>Enjoy a seasonal take-and-make felt kit for December! Celebrate the start of winter and select one of three project options: Red Cardinal, Gingerbread House, or Knotted Tree. Each kit provides the pattern, directions, and materials to create and enjoy a festive felt craft. </p>\r\n\r\n<p>Pick up a free craft kit at the Library&rsquo;s Information Desk any day this month! One kit per visit, please; available while supplies last.</p>",
  "more_info": "<p>Free printable <a href=\"https://guides.sspl.org/FeltFun\">directions and patterns for &ldquo;Take and Make&rdquo; kits</a> are always available online for current and past projects.</p>",
  "url": "https://sspl.libcal.com/event/11438923",
  "admin_url": "https://sspl.libcal.com/aevent.php?c=10655&m=edit&event_id=11438923",
  "location_id": "0",
  "location": "",
  "audience_id": "1604|1623|1627|1628",
  "audience": "Children|Adults|Teens|All Ages",
  "cat_id": "64132",
  "category": "Art, Music, and Movies > Crafting",
  "owner": "Kali Nagler",
  "presenter": "Miriam Meier & Jasmine Gage",
  "cal_id": "10655",
  "calendar": "Library Events",
  "color": "#B3C8EF",
  "image": "https://lcimages.s3.amazonaws.com/data/feat_img/4700/10655/1697730913.png",
  "geo_id": "ChIJh9RRqdA53okR1XeeOfmgCng",
  "geo_lat": null,
  "geo_long": null,
  "cost": "0",
  "registration": "0",
  "registration_open": "0",
  "seats": "",
  "seats_taken": null,
  "physical_seats": null,
  "physical_seats_taken": null,
  "online_seats": null,
  "online_seats_taken": null,
  "zoom_email": null,
  "online_user_id": null,
  "online_meeting_id": null,
  "online_host_url": null,
  "online_join_url": null,
  "online_join_password": null,
  "online_provider": null,
  "wait_list": "0",
  "future_dates": null
}]
BrendonKoz
Advocate I
Advocate I

In trying to put together a proof-of-concept to run through the data, I did notice that Python can't read null as an object literal value, and it must instead be "null" (a string value). I was unable to determine how to properly import the json module in PAD, so handled it with a REGEX text replacement. Now the proof-of-concept is running, but returning an empty list without error when the values obtained for the strings in the test case were obtained from the live example, so I'm kind of uncertain how/where the error(s) are coming from with the string values... I might need the Python json module, but can't figure out how to import it. Running the flow with simple test-case values works as expected (without complex string values, or the imported json module), and without error, so there's definitely something in the live variables' values that are likely not escaped properly.

 

Working PAD Flow:

 

SET JsonEvents TO $'''{\"events\":[{\"id\":\"11438923\",\"title\":\"Event One\"},{\"id\":\"11362898\",\"title\":\"Event Two\"},{\"id\":\"10734368\",\"title\":\"Event Three\"}]}'''
SET HandledEventsTable TO $'''[{
  \"id\": \"11438923\",
  \"allday\": \"1\",
  \"multiday\": \"1\",
  \"private\": \"0\",
  \"start\": \"1701406800\",
  \"end\": \"1703393999\",
  \"setup\": \"0\",
  \"breakdown\": \"0\",
  \"title\": \"Festive Fun with Felt: December Take-and-Make Craft Kits\",
  \"description\": \"<p>Enjoy a seasonal take-and-make felt kit for December! Celebrate the start of winter and select one of three project options: Red Cardinal, Gingerbread House, or Knotted Tree. Each kit provides the pattern, directions, and materials to create and enjoy a festive felt craft. </p>\\r\\n\\r\\n<p>Pick up a free craft kit at the Library&rsquo;s Information Desk any day this month! One kit per visit, please; available while supplies last.</p>\",
  \"more_info\": \"<p>Free printable <a href=\\\"https://guides.sspl.org/FeltFun\\\">directions and patterns for &ldquo;Take and Make&rdquo; kits</a> are always available online for current and past projects.</p>\",
  \"url\": \"https://sspl.libcal.com/event/11438923\",
  \"admin_url\": \"https://sspl.libcal.com/aevent.php?c=10655&m=edit&event_id=11438923\",
  \"location_id\": \"0\",
  \"location\": \"\",
  \"audience_id\": \"1604|1623|1627|1628\",
  \"audience\": \"Children|Adults|Teens|All Ages\",
  \"cat_id\": \"64132\",
  \"category\": \"Art, Music, and Movies > Crafting\",
  \"owner\": \"Kali Nagler\",
  \"presenter\": \"Miriam Meier & Jasmine Gage\",
  \"cal_id\": \"10655\",
  \"calendar\": \"Library Events\",
  \"color\": \"#B3C8EF\",
  \"image\": \"https://lcimages.s3.amazonaws.com/data/feat_img/4700/10655/1697730913.png\",
  \"geo_id\": \"ChIJh9RRqdA53okR1XeeOfmgCng\",
  \"geo_lat\": null,
  \"geo_long\": null,
  \"cost\": \"0\",
  \"registration\": \"0\",
  \"registration_open\": \"0\",
  \"seats\": \"\",
  \"seats_taken\": null,
  \"physical_seats\": null,
  \"physical_seats_taken\": null,
  \"online_seats\": null,
  \"online_seats_taken\": null,
  \"zoom_email\": null,
  \"online_user_id\": null,
  \"online_meeting_id\": null,
  \"online_host_url\": null,
  \"online_join_url\": null,
  \"online_join_password\": null,
  \"online_provider\": null,
  \"wait_list\": \"0\",
  \"future_dates\": null
}]'''
Text.Replace Text: JsonEvents TextToFind: $'''([^\"\"])(null)([^\"\"])''' IsRegEx: True IgnoreCase: True ReplaceWith: $'''$1\"null\"$3''' ActivateEscapeSequences: False Result=> JsonEvents
Text.Replace Text: HandledEventsTable TextToFind: $'''([^\"\"])(null)([^\"\"])''' IsRegEx: True IgnoreCase: True ReplaceWith: $'''$1\"null\"$3''' ActivateEscapeSequences: False Result=> HandledEventsTable
@@copilotGeneratedAction: 'False'
Scripting.RunPythonScript PythonCode: $'''events = %JsonEvents%
handled = %HandledEventsTable%
events = events[\'events\']

# Extract the ids from the second object
handled_ids = set([h[\'id\'] for h in handled])

# Filter out duplicates from the first object
events = [e for e in events if e[\'id\'] not in handled_ids]

print(events)''' ScriptOutput=> PythonScriptOutput ScriptError=> ScriptError

 

 

 

BrendonKoz
Advocate I
Advocate I

Okay, now it's working with the live values if they're assigned manually in my test case. 🤦‍♂️ Compilers are fun when they work differently the next day/session. I'll have to try converting null values to a string "null" in the live version (tomorrow) and see what happens.

Agnius
Most Valuable Professional
Most Valuable Professional

If you want to import a module that is not installed with the PAD package (i.e. when you try to import and it doesn't find it), you need to install Python to your machine and then provide the path to the modules directory to the Run Python script action. This will then allow you to import other modules. 

 

But be aware that this only works with Python 2.x versions. PAD does not natively support Python 3 scripts.

-------------------------------------------------------------------------
If I have answered your question, please mark it as the preferred solution. If you like my response, please give it a Thumbs Up.

I also provide paid consultancy and development services using Power Automate. If you're interested, DM me and we can discuss it.

-------------------------------------------------------------------------------------------------------------------------
If I have answered your question, please mark it as the preferred solution. If you like my response, please give it a Thumbs Up.
Regards, Agnius Bartninkas
BrendonKoz
Advocate I
Advocate I

In the end, I'm not 100% sure what worked to get around my initial issue; it never appeared again. @Agnius' tip about how to reference Python modules was exactly what I needed to know to properly import modules (I wrongly assumed since PAD supposedly used IronPython, that it was already preinstalled somewhere.) I was unable to get PAD's Python to properly convert the JSON string into a Python object; it might've been that my formatting of the string was wrong, or there was something else that I was missing...either way, I found a trick to handle nulls that simply declared `null` as a variable, and assigned it Python's null value (`null = None`) which solved my null value issue in converting the JSON to a Python object (they're so similar, it just became an object, essentially).

 

 

 

 

import json

null = None

events = %JsonEvents%
handled = %HandledEventsTable%
events = events['events']

# Extract the ids from the second object
handled_ids = set([h['id'] for h in handled])

# Filter out duplicates from the first object
events = [e for e in events if e['id'] not in handled_ids]

print(json.dumps(events))

 

 

 

 

After exporting the string back to PAD via the `print` statement, I noticed there was a BOM character preceeding the intended value. In order to strip this, I used a simple find/replace REGEX, primarily because I couldn't figure out how to prevent the BOM from being returned from within the Python code itself.

 

Text.Replace Text: PythonOutput TextToFind: $'''^.*\\[''' IsRegEx: True IgnoreCase: True ReplaceWith: $'''[''' ActivateEscapeSequences: True Result=> PythonOutput

 

NOTE: PythonOutput is the variable name that the "Run Python script" action created in this example.

 

After that, I converted the JSON string back to a custom object for PAD to continue doing what it needed to do with my list of custom objects.

 

Definitely a hacky solution that I'm not 100% proud of, but sometimes we just have to move forward and don't have time to look back. I still don't know why I was having string quote issues with my first attempts, and since that was never really solved, I'm not sure if marking any response here as a solution is appropriate, but I wanted to follow up and at least provide what my solution ended up being in case it helps someone else in the future.

 

Additional notes:

  1. PAD doesn't (currently) handle null values as provided initially from a JSON string all too well. If accessing a property that has a null value, PAD will error and say the property doesn't exist (in fact its value doesn't exist). To workaround that, change the value of `null` to an empty string.
    BrendonKoz_0-1702407662482.png
  2. Python, as how my code is written above, will double-escape backslashes, which were inserted to escape forward slashes. During conversion back, this causes the string to remain escaped (ex: http:// would instead be http:\/\/), so before converting from a JSON string back to a custom object in PAD, I had to replace "\\" with "\" (non-regex).
  3. Be aware with any/all string replacements that contains external content; you might unintentionally replace something you didn't intend to. The word part of "null" might've actually been intended within usable textual content. In my case that is unlikely, but your case might be different. If that's the scenario, your find/replace might need to be different in order to account for that.
Agnius
Most Valuable Professional
Most Valuable Professional

Thanks for sharing your solution. I'll take the liberty of marking it as the solution to the topic.

-------------------------------------------------------------------------------------------------------------------------
If I have answered your question, please mark it as the preferred solution. If you like my response, please give it a Thumbs Up.
Regards, Agnius Bartninkas

Helpful resources

Announcements

Check out the Copilot Studio Cookbook today!

We are excited to announce our new Copilot Cookbook Gallery in the Copilot Studio Community. We can't wait for you to share your expertise and your experience!    Join us for an amazing opportunity where you'll be one of the first to contribute to the Copilot Cookbook—your ultimate guide to mastering Microsoft Copilot. Whether you're seeking inspiration or grappling with a challenge while crafting apps, you probably already know that Copilot Cookbook is your reliable assistant, offering a wealth of tips and tricks at your fingertips--and we want you to add your expertise. What can you "cook" up?   Click this link to get started: https://aka.ms/CS_Copilot_Cookbook_Gallery   Don't miss out on this exclusive opportunity to be one of the first in the Community to share your app creation journey with Copilot. We'll be announcing a Cookbook Challenge very soon and want to make sure you one of the first "cooks" in the kitchen.   Don't miss your moment--start submitting in the Copilot Cookbook Gallery today!     Thank you,  Engagement Team

Announcing Power Apps Copilot Cookbook Gallery

We are excited to share that the all-new Copilot Cookbook Gallery for Power Apps is now available in the Power Apps Community, full of tips and tricks on how to best use Microsoft Copilot as you develop and create in Power Apps. The new Copilot Cookbook is your go-to resource when you need inspiration--or when you're stuck--and aren't sure how to best partner with Copilot while creating apps.   Whether you're looking for the best prompts or just want to know about responsible AI use, visit Copilot Cookbook for regular updates you can rely on--while also serving up some of your greatest tips and tricks for the Community. Check Out the new Copilot Cookbook for Power Apps today: Copilot Cookbook - Power Platform Community.  We can't wait to see what you "cook" up!    

Welcome to the Power Automate Community

You are now a part of a fast-growing vibrant group of peers and industry experts who are here to network, share knowledge, and even have a little fun.   Now that you are a member, you can enjoy the following resources:   Welcome to the Community   News & Announcements: The is your place to get all the latest news around community events and announcements. This is where we share with the community what is going on and how to participate.  Be sure to subscribe to this board and not miss an announcement.   Get Help with Power Automate Forums: If you're looking for support with any part of Power Automate, our forums are the place to go. From General Power Automate forums to Using Connectors, Building Flows and Using Flows.  You will find thousands of technical professionals, and Super Users with years of experience who are ready and eager to answer your questions. You now have the ability to post, reply and give "kudos" on the Power Automate community forums. Make sure you conduct a quick search before creating a new post because your question may have already been asked and answered. Galleries: The galleries are full of content and can assist you with information on creating a flow in our Webinars and Video Gallery, and the ability to share the flows you have created in the Power Automate Cookbook.  Stay connected with the Community Connections & How-To Videos from the Microsoft Community Team. Check out the awesome content being shared there today.   Power Automate Community Blog: Over the years, more than 700 Power Automate Community Blog articles have been written and published by our thriving community. Our community members have learned some excellent tips and have keen insights on the future of process automation. In the Power Automate Community Blog, you can read the latest Power Automate-related posts from our community blog authors around the world. Let us know if you'd like to become an author and contribute your own writing — everything Power Automate-related is welcome.   Community Support: Check out and learn more about Using the Community for tips & tricks. Let us know in the Community Feedback  board if you have any questions or comments about your community experience. Again, we are so excited to welcome you to the Microsoft Power Automate community family. Whether you are brand new to the world of process automation or you are a seasoned Power Automate veteran - our goal is to shape the community to be your 'go to' for support, networking, education, inspiration and encouragement as we enjoy this adventure together.     Power Automate Community Team

Hear what's next for the Power Up Program

Hear from Principal Program Manager, Dimpi Gandhi, to discover the latest enhancements to the Microsoft #PowerUpProgram, including a new accelerated video-based curriculum crafted with the expertise of Microsoft MVPs, Rory Neary and Charlie Phipps-Bennett. If you’d like to hear what’s coming next, click the link below to sign up today! https://aka.ms/PowerUp  

Tuesday Tip | How to Report Spam in Our Community

It's time for another TUESDAY TIPS, your weekly connection with the most insightful tips and tricks that empower both newcomers and veterans in the Power Platform Community! Every Tuesday, we bring you a curated selection of the finest advice, distilled from the resources and tools in the Community. Whether you’re a seasoned member or just getting started, Tuesday Tips are the perfect compass guiding you across the dynamic landscape of the Power Platform Community.   As our community family expands each week, we revisit our essential tools, tips, and tricks to ensure you’re well-versed in the community’s pulse. Keep an eye on the News & Announcements for your weekly Tuesday Tips—you never know what you may learn!   Today's Tip: How to Report Spam in Our Community We strive to maintain a professional and helpful community, and part of that effort involves keeping our platform free of spam. If you encounter a post that you believe is spam, please follow these steps to report it: Locate the Post: Find the post in question within the community.Kebab Menu: Click on the "Kebab" menu | 3 Dots, on the top right of the post.Report Inappropriate Content: Select "Report Inappropriate Content" from the menu.Submit Report: Fill out any necessary details on the form and submit your report.   Our community team will review the report and take appropriate action to ensure our community remains a valuable resource for everyone.   Thank you for helping us keep the community clean and useful!

Users online (4,818)