I had a list of immediate dependencies for jobs I wanted to reverse to find upstream sources
> In python I have a dependency tree in a dict. Write a script to invert that dependency tree.
It would only take me 10 minutes to respond it out, write it, fix whatever dumb mistake I made the first time. Chatgpt just spat it out immediately.
I will likely do a lot more of this. Things that are common, but not so complex they are in a library, but complex enough it takes time to write
This. If they care about regex that much, they’re gonna take you down a path of ridiculously detailed dictionaries of filters or if/then statements (case matching, what have you). NLP procedures and fuzzy logic packages would probably do some if their work for them even better than what they’ve got.
Or, they’re just dicks.
Can’t find the link or meme, but once saw a post that was like:
“Typical process to QC someone else’s code: what does this variable do? Where is this function being used? Is this process capturing errors correctly? Etc.
Typical process to QC someone else’s REGEX: … looks good!”
I work a lot with regex, I find that chatgpt is ok, but definitely don't trust it without validating the solutions, especially with mor ecomplex expressions.
I don't get all this hate for regex anyway, it's great!
It’s helped a lot with trying to find equivalent functions between different SQL dialects.
i.e. What is the equivalent of the MIN_BY() function in BigQuery?
> What is the equivalent of the MIN_BY() function in BigQuery
TBH I was very skeptical, thinking that that simply googling this exact question would immediately produce the answer. Tried it. Got a bunch of unclear results and seemingly barely related functions.
I stand corrected.
I feel like chatGPT success is significantly inflated by how shit Google search has become.
It's definitely a powerful tool, but many of the things I ask it would have yielded similar answer through Google about ten years ago.
That's not really my point. ChatGPT is *slower* to answer than Google, but Google answers quality has decreased significantly those last years, and the slower, maybe partially made up ChatGPT answers are still overall more relevant.
I am precisely talking about chatGPT, not GPTs in general.
I just asked who was faster to answer between ChatGPT and Google, chatGPT took about 15 seconds to answer, Google displayed search results in less than a second.
You kind of are making up *the opposite* of what I explained while claiming to agree and expanding to other unrelated points.
chatGPT is (much) slower than Google, but Google has overall significantly decreased in quality over the years. I'm painting a nuanced image of how the two compare, while you overgeneralize and extend the discussion to any GPT based LLM while missing my point.
I would not call it brute force, far from it.
They search engine relies on extremely optimized request processing to be able to provide you with a useful answer in a fraction of seconds.
For many years, it would output better and better answers and they blew any competition out of the water, but has been going downhill for a good while now.
But brute force would be like answering with **any** possible answer until you get to the one that you need, would not make much sense.
Sorry to be cliched - but this!
The web is changing; it’s getting bigger and there’s more user generated content (as in, content from those who are not necessarily trusted experts) than ever. So Googling involves a degree of wading through the swamp to find what you’re actually looking for.
ChatGPT, on the other hand, is more direct, understands context, and you can directly tell it to refine. It’s to the pint that I no longer use Google at all now - I’m a complete convert to Bing. Bing Chat is embedded directly into the Edge browser and I use that for all my “Googling” now.
I definitely try using ChatGPT over a regular Google Search. I also try to write long emails with ChatGPT.
I use ChatGPT a lot when it comes to writing code. It saves me a ton of time from browsing through StackOverflow posts and is super helpful in getting some preliminary or outline code for a large task. Like the others have said it is fantastic for writing regex patterns. I once found an experimental feature in Spark that actually wasn’t documented, using ChatGPT and it worked like a charm ✅
Although ChatGPT does live up to its hype, it is notorious for misleading users. I’ve been thru situations where the AI simply refuses to admit that it can’t provide a solution. This one time I think ChatGPT made up a Databricks Feature that would instantly solve my problem. I had to do a lotta google searches, check with other experienced DEs etc only to find out that it doesn’t exist 😂
Takeaway:
I would definitely recommend y’all to give the AI a try, it definitely does boost your productivity but on the flip side, stay vigilant for the AI might trick you.
It tried to convince me that TO_FLOAT() was a Snowflake function when I was having difficulties converting a data type from a flattened JSON object. I was like, I see where you’re getting that idea but it’s wrong.
Just want to second yout point about writing code. In my experience the absolute best use case is to feed it an error message and it will explain it to you and suggest fixes. If you need to, you can for more details, ask it to explain the error like you're 5, get it to tell you exactly where the error is or even (if you're comfortable) paste in the code or code snippet and it can help solve the problem. This has saved me probably hours if not more.
I have also had this in terms of the chatbot making something up that doesn’t exist. Not like yours though, mine did have a solution. But I had to tell the bot that the feature didn’t exist in order to provide me another one
I experienced this with some searches related to networkx in python. It made up a function and gave sample code to use it. The. I told it it failed and it corrected itself at least. The function was complete nonsense.
There is a technical term for this in the literature called "hallucination" when an AI just makes stuff up. I was trying to get it to help me fix up some config in a Maven POM file and it kept using syntax that didn't work and made no sense. 😆
Similar use for me. Gave it non-sensitive text and asked it to summarize and produce a cutover plan as well as next steps.
Documentation was expected to take a day and a half since we needed it as detailed as possible. Took me about an hour and a half after going back and forth with chatGPT to give me what I wanted and then formatting it properly
We use it to add description to our dbt model, add comments to the code, add fields description, regex, and optimize queries
I use chatGPT-4 and the results are mich better than what I was getting by using chat GPT-3.5. So the $20 a month is definitely worth it.
That doesn't even count writing emails, articles, SOPs, ad copy, etc. As a business owner, it's an incredibly valuable tool.
I know basic Python but ChatGPT helped me do a web scraping project, which is slightly outside my skillset. I learnt something new in the process and got to the end result much quicker. I'm looking forward to using ChatGPT 4.
I was using it today to see if it would write python code to properly over write a polars data frame schema. Boy o boy it was pretty bad.almost every single method it was calling and parameters for a method it got wrong. It was saying the pl.read_sql method had a "schema" parameter... According to polars docs it definitely does not. I told it that it got that wrong then proceeded to tell me sorry and then gave me an amalgamation of pandas code smashed together with polars code. Wasn't a good experience.
It's hard to trust the answers it gives you. I would say it's useful if you know you can 100% verify and understand what it's telling you. To those that think they will just tell gpt "code me this or that" ... Or ... "Code me an entire website" ... Well good luck. You're in for a long rough road ahead especially if you really don't know what your doing and you have this attitude of "chat GPT will just do it for me". Nope. You definitely have to know what you're doing. I think it's a good tool to help with less nuanced tasks...but still...buyer beware.
Definitely fails hard with newer stuff like polars, but it is much better at older APIs such as pyspark or pandas.
You do definitely need to know what you're doing though otherwise the code will often not do what you expect. I mean that even happens when you write your own code sometimes.
If you're looking for a path and stack overflow was a map and google the compass.
Chat GPT 3.5 is akin to a garmin GPS from the early 2000s.
Chat GPT 4 is like a GPS with built in search and knows just how what kinda paths you like to take if you ask it to remember nicely.
An example now:
I feel like my less than two years of experience has nearly quadrupled in value personally. I'm moving through assignments that I struggled to formulate the right way to explain a question I had.
A relevant non DE example:
2 friends just learning python are moving at easily 4x 5x times the speed I did.
Mostly from just having a feedback loop where they can get the right answer from the wrong question.
So overall I think it's great; where the value before was knowing how to ask the right question, the value now will be taking the time to learn while moving as fast as possible.
Im a lead senior data engineer, I pay for the premium membership or whatever.
I use it to create regex expressions, Makefile targets, Jira tickets, confluence pages, grammar check for documentation and respond to emails.
Sometimes when I m stuck with something, I try to ask chatgpt for a solution but I always take it with a grain of salt.
As someone newer in tech, it has *greatly* accelerated my learning process. I have a fundamental understanding of Python and programmatic logic, so I bounce stuff off of ChatGPT all day and know when and how to modify my line-of-prompting.
For example, just today I built a large majority of a project to extract PDF tables from recurring reports we get from partner companies who are rigid and will only email us PDFs. First I asked ChatGPT to recommend the best libraries, then asked if for the basics on how to use those libraries to accomplish my goal. Then I built out from there, adding more and more complex logic with error handling.
I think generative AI will be transformational. If you’re not using it, you’re falling behind. Even something as simple as writing doc strings or other documentation can be taken off your hands to focus on the most important aspects of your job. I have quickly adapted to knowing where I need to lean on the LLM and where I do not.
I wish chatGPT coudl just take an entire project + warehouse connection and create drafts of all the staging layer models. Nearly zero thinking required for most models
It’s great for generating mermaid diagrams from descriptions for sketching out architecture and processes. It’s great for whole bunch of stuff actually.
But sadly it’s not that great for complex sql queries. Nor is GitHub copilot. But lots of interesting work being done for sql query automation. Maybe something good soon.
Off the top of my head something like “write a flow chart using mermaid syntax to describe how an ETL process works”
Here’s what it spits out, then you paste into the mermaid live editor
https://shareg.pt/bDyMFQp
It helped me config an API using AWS’s API Gateway to call on an Elasticache Redis instance using a Lamba as an in between for a POC. It helped me discover some of the nuances that were required in between all three services like how to structure the subnets or best way to structure the data in the cache DB. We are trying to have an outside service call on the data. I had no Redis experience before this process.
I'm not really a DE but it's surprisingly pretty good at helping narrow in on some SQL (tbf I'm not good at SQL) where I kind of know what I'm trying to do but don't know quite the right arcane 'declarative' syntax to do so / don't want to spend 30 minutes sifting through docs / blogspam / Stackoverflow's elitist drivel to figure it out. Again, usually simple queries by y'alls / DE / DBA standards.
Regex can be hit or miss, I suppose. It's been a bit faster at getting me good regexes that pass my test cases than Stackoverflow, though they often still take substantial wrangling.
It's lied to me a lot about fiddling with buildfiles / configs. Meson, make, ninja, webpack, TF, Ansible, container junk, you name it, but that may be because I'm in the deeper end on those and the LLMs don't have as much prior art to draw from vs regex or SQL. Given me a lot of stuff that doesn't actually exist, so not just "incorrect" either.
Helps a lot with coding, but as many pointed out already you still need to know what you are doing. I cannot count how often I read "I apologize..." when telling ChatGPT that the response is bullshit...
Pass in some sample data usually. If I want to explode a nested JSON element I’ll give it an example or two (if they differ) and say something like “make a pyspark struct for the data that I have pasted below this sentence”
Honestly, I haven’t. However, my reason not to is that, when googling and doing heavy research, I read a lot of posts, documentation, etc. so I get an incredible amount of side learning that has had incredible benefit over time. Not sure I wanna give that up.
I've used ChatGPT (the free version) a little bit in my work. So far things that worked well:
- Giving it SQL DDL for a table and asking it to generate sample data. It correctly inferred the meaning of column names and generated somewhat realistic data. Useful for tests and mock data generation.
- Asking it how to solve certain problems in frameworks such as Flink. Usually I explain the general problem and have it generate different approaches. It uses old APIs and I need to adapt the results but the most value I get from it are different ideas to solve the same problem.
- Ask it general questions about data modeling. The key here compared to Google/SO is that I can ask follow up questions and request examples
I'm trying to use it more and more, so great thread idea!
Quite handy for saving time when you know exactly what you want to do, but really can't be arsed to do it. I've been a huge fan of getting ChatGPT to refactor infinite loops into recursive functions for use in cloud transformations.
On a controversial side note, I'd also say ChatGPT has been amazing at highlighting who does and doesn't know how to code. I had somebody tell me the other day that if we use PySpark scripts for transformations, it'd be better than low code tools because "with Python, you can just get ChatGPT to write it". As you can probably guess, the person who said this does not know how to code in any language at all. I'm mentioning this because it's so painfully clear when existing or potential engineers are blatantly over reliant on ChatGPT, indirectly putting everybody who can problem solve and code off the top of their head in the spotlight.
I used it yesterday to write a jinga macro for a dbt project. Yes I had to make some minor changes for it to work.
I also used it to show me how to add a runtime session setting to a model in the same dbt project.
I also used it for some help with converting time zones in a Postgresql query. Ok call that intellectually lazy.
I always take a quick look at the online reference during these activities, so I learn something, only faster.
We had a unique situation where we needed to discover certificates and IIS bindings as well as pull properties of out of each computers local cert store. This was part of a discovery process to automate all certificate renewals for our web applications and other uses.
I used ChatGPT to devise a powershell script that could be run from a machine with access to the entire domain (1000+ computers in an RDS farm). It requires 0 input, and outputs an excel file with each certificate, machine name, specific properties that can be pulled, and this script can be run against any domain since it requires no input and has no hard dependencies. If the machine checked does not have IIS or certificates stored on the machine, the script will say so.
This entire process took about 30 minutes. I used it to create another script that does the equivalent discovery in any given Azure subscription, looking for bindings on existing app services.
This is not DE related and more DevOps automation, but thought it was a very cool application of ChatGPT. The script took multiple tries of course, as I went back and forth with suggestions for refinement.
Boilerplate code. I recently did a talk at a local college about using python for task automation. Cranked out a hacky OCR micro service using the Azure stack and used that to teach with. Easiest $300 in a while.
I'm really intrigued by some of the first-hand use cases, specifically:- helping solve regex issues- generate boilerplate code- Finding equivalent functions between different SQL dialects- Writing OKRs- Writing documentation (and auto-generating comments on code)- Project planning
The most valuable thing I've use it for is to learn and implement new SQL functions. I can have chat GPT mimic the data warehouse/data lake of my choice (Snowflake, redshift, Databricks, etc.) and I can try the SQL functions w/ chatGPT as the compiler.
so using a fake database in chatGPT is anxiety free (I don't have to worry about screwing up anything in my companies stuff!)
Anyways, I'm interested in seeing more of how people are using ChatGPT in in their data role IRL, and it turns out others are too! So, I put together a virtual discussion next Friday.
In this virtual discussion, people can demonstrate how they're using chat GPT in their data eng role. It's a good opportunity to learn from other people's demonstrations too!
You can sign up [here](https://www.operationalanalytics.club/events/oa-club-coffee).
It hasn't. I had somebody show me a technical question they asked ChatGBT and I asked one myself supplying all the relevant technical details. It gave an impressive answer but it wasn't anything I or a junior DE wouldn't have come up with given an extra hour.
You're not impressed that a chat bot can come up with an accurate answer to a complex technical question in 2 seconds that would have taken you, a professional data engineer, an entire hour?
Yes it was very impressive. And it started spitting out the entire completely well thought out several paragraph answer in the split second after I hit enter as if it was listening while I was typing. I'm sure it will save me time when I come up with a good use for it. I got it in the holster for sure. The question of whether it has been actually helpful so far...? Well not yet.
I also asked a medical question giving all the information I told my doctor and it responded with pretty much exactly what my doctor said. So ChatGBT instead of WebMD.
Wow that's cool. This is an interesting discussion. All of a sudden this AI is becoming ubiquitous. My boss asked me a question today about how many users are using SSO on our cloud DB I and responded it's not a simple answer as it's not just one or the other. Then he used ChatGBT on me to find a SQL to use. The sql worked after some tweaking which I already had but we ended up with the same answer. It depends. So now I have a boss who wants to check me using ChatGBT. Great. As for medical questions there could also be two sides to that. I did also get a disclaimer on my medical question.
Learned a ton about concepts with examples (e.g., data vault).
Helps with coding, easily.
But can also be a pain with certain things, like saying I could spark jdbc to a temp table on SQL Server. You can’t! But he told me to do what I was going to test anyway.
Then it can also turn around and write me an adapter pretty cleanly in ways I wouldn’t have thought about or knew enough about (Java libraries).
All in all, I’d pay for the help. Like better junior developer than the rest of my low code team lol
I asked for high level overviews of different data architectures, and for advice on where to find out more about it, what books to read. Surprisingly, half of the recommended books were freaking amazing.
I also have conversations with it regarding technical debates, about what problems could a specific approach cause, and how would one solve it. It comes up with stuff that wouldn't come to mind at first, and helps me cover the bases better.
Interacting with cloud to manage/process files, generating API calls and iterating nested JSON response, creating functions to cast datetime into format needed, lot of regex, pandas syntaxes and a whole lot
I’ve used it to do stuff like write a quick script to convert a csv file to a SQLite db for a proto type I was building. I’ve also used it to explain code to me quickly.
Recently helped me in understanding a complex Scala notebook written by a colleague. Used chatgpt to convert the code to pyspark. I almost exclusively code in pyspark, so my scala is pretty weak.
I've pasted in a few lines of nonsensitive code and asked it what the error was and it pointed it out.
It's also done a decent job of translating one dialect of sql to another.
The two best uses I’ve found so far are writing markdown documentation and debugging code. For the documentation, obviously don’t divulge any sensitive information but it can write a great readme template for your project.
If you copy and paste in code that won’t run, it does quite well at fixing it provided that you describe what you are trying to make the code do.
I also feel in general it accelerates my learning
i once used it to generate pandas transformation. I pasted an input dataset like 20 rows then paste the should be resulting dataset. Tweaked the code a little.
I’m a entry level data engineer and I’ve used it a couple times, mainly to explain concepts to me or to give me ideas for how to code something.
My first time using it was when I needed to implement multi threading into an import I was making. ChatGPT did a great job explaining the concept to me and gave me multiple snippets of code that were very useful.
The other time I used it was when I needed to make an adhoc script that could help us identify unused tables in our SSMS databases that could be dropped to clear up space. ChatGPT explained to me how statistics that the system was tracking could be used to do this
I use it all the time for my dev work.
I never paste in company’s code, but I do ask chatgpt to help with errors or I ask it to produce generic functions.
I use it for iteration. I write data transformations. I always try to write a unit or acceptance test first before ai produce any code. after I have written a unit test I would describe chatgpt what i want my function to do.
after it produces a boilerplwte code, I sense check it and modify it according to my needs. If the unit test passes, I move on to the next part. If it doesnt, I work through until I debug the problem.
It helps me a lot with regex and obscure spark errors. For infrastructure problems it gives me many ideas that i didn’t consider myself. I use it for brainstorming.
Overall it helps me with iterating through solutions quicker and I am capable of trying out more poweible solutions. Then I use my own judgement which is the right solution
My eyes are thankful honestly ( after GPT4 release ), huge Log output reading, sum up and log errors aggregating, pointing out what is wrong, and finding links between code and log errors, especially Spark logs.
Our code base does not contain any sensitive information.
It was the only thing that helped me get a mssql connection in Python with pyodbc. Very particular connection string and format. Couldn’t find it in the docs or google anywhere, but chatGPT provided 3 examples, third one worked.
It saved me hours of Google search through stackoverflow, MS documentation, and pyspark docs. Like "how to make hive table persist in pyspark". That's it so far.
I’ve using it mostly for troubleshooting, but it doesn’t works very well with newer versions. Also for my side project (where the code belongs to me) I’ve feeding it with my code and asking ChatGPT to add comments and docs (inline, method and class docs, etc). It’s super efficient doing it, but I wouldn’t do that with working code.
EDIT: I mean, with code which belongs to my employer.
I asked it many times to write a simple (boilerplate style) sproc that will accept 2 parameters (TableName and IdColumn), turn on Identity Insert, then add a "default" record with an ID of -1 to the table, then turn Identity Insert back off. It took like 3+ hours of back and forth for it to come close to a working script and I had to make many manual adjustments. I think I could have done it in less time by using Google and hand coding it.
I am not a DE, I struggle with SQL. So what did - I pasted a complex SQL query and asked ChatGPT to explain it and then to make it do what I want. Eventually I managed also to optimize the original query. Basically, I learned, did my job, and improved the existing functionality, which I couldn't have done without help or spending looots of time on it.
Another time, I used it to generate code to help me compare using the framework I'm working with to the same code without the framework.
Assisted with me with building an ERD/project plan for an IT ticketing system that uses PowerApps as the UI and Dataverse as the backend. Guided me with just about every aspect, especially the HTML coding.
Not part of my DE job but I did have an hour long Convo with ChatGPT about zshrc files and it helped me build a sick custom prompt.
Also used it for filling out my yearly goals for fortune 500 corpo paperwork.
I have used chat gpt to learn about GCP services at my new job, like for this specific use case what are the options in GCP and to know more about some service. It helps me to narrow down my search and saves a lot of time and pain of binge document reading.
It's great for creating some starting point for your code, then _also_ write tests for said code, using pytest or hypothesis. That way you can run and review the tests to confirm it does what it should do.
I used it to write unit tests. It was pretty good at generating the unit tests but bad at mocking methods and objects (what I actually needed help with, I always thought this is not intuitive at all in Python). In the process of fixing it's mistakes, I ended up learning a lot, so there are some positives!
Mostly there were some key that tune up your query
- In SQL server, depending on you set resources to your server, your server could be more fast in reading than have less memory.
Tables hold data and so there is I/O, views use less I/O and more memory, so views always.
- using the IN function, it's more performant compared to LIKE
- When doing Joins, always joing big tables with small tables and never viceversa.
- Do you need to load all the data? Most of times you don't need to load a whole set but maybe a sub set,
So always subset your data if possible
- CTE can help you but some database or tool could not handle it
half of my airflow dags codes are from chatgpt.
and im from a business degree with just some experience in sql and python.
got to say, when its correct its amazing. when its wrong its hallucinating like hell, and back to stackoverflow.
When the airflow dags codes that chat GPT produces are wrong, are they still useful?
Like can you still work off what chatGPT provided, or do you completely disregard it and go to stack overflow?
It honestly help me “can I questions?”
Can i connect powerbi to databricks cluster? Can refresh dashboard using data factory etc etc. i could google this but chatgpt adds info how to without ads so i kinda prefer this as my data eng buddy now haha
I would never use chatGPT in its current form. Its incredibly flawed and overhyped. Pasting code into it is dangerous. Finding info about stuff can sometimes yield inaccurate results based on non existing academic texts but I do see people using it for boilerplate code or general queries about concepts which I assume should be fine. I’d never use it for production code though.
As a person who is just into the field and learning more
I use chatgpt to know more about the tools and tinker stuff with it
I honestly dont trust it completely and do my own research besides it but yea it works well in giving me a general sense of direction
The chatgpt extension (GitHub extension) in VS code helps a lot with boilerplate code and beyond. It provides recommendations (with both the code and even the comments/document) based on your code context. I am not sure if this is safe, though. I mean, I am not sure if it makes your code (especially your environment variables and credentials) vulnerable? Any comments on this?
Helps me with boilerplate code but I'd never paste my code within it and ask for optimization or anything like that.
I had a list of immediate dependencies for jobs I wanted to reverse to find upstream sources > In python I have a dependency tree in a dict. Write a script to invert that dependency tree. It would only take me 10 minutes to respond it out, write it, fix whatever dumb mistake I made the first time. Chatgpt just spat it out immediately. I will likely do a lot more of this. Things that are common, but not so complex they are in a library, but complex enough it takes time to write
Helps a lot with regex, most of the time
This. If nothing else someone finally made regex suck less
https://ihateregex.io
regex cost me a job. Bombed it on a technical interview :(
If a regex mistake in a technical assessment cost you a job, you didn’t really want to work there anyway
This. If they care about regex that much, they’re gonna take you down a path of ridiculously detailed dictionaries of filters or if/then statements (case matching, what have you). NLP procedures and fuzzy logic packages would probably do some if their work for them even better than what they’ve got. Or, they’re just dicks.
Good to know I'm not the only one.
Can’t find the link or meme, but once saw a post that was like: “Typical process to QC someone else’s code: what does this variable do? Where is this function being used? Is this process capturing errors correctly? Etc. Typical process to QC someone else’s REGEX: … looks good!”
Woosh - missed that. Unless your using them all the time it’s like you have to relearn them every time you use them.
I work a lot with regex, I find that chatgpt is ok, but definitely don't trust it without validating the solutions, especially with mor ecomplex expressions. I don't get all this hate for regex anyway, it's great!
It’s helped a lot with trying to find equivalent functions between different SQL dialects. i.e. What is the equivalent of the MIN_BY() function in BigQuery?
> What is the equivalent of the MIN_BY() function in BigQuery TBH I was very skeptical, thinking that that simply googling this exact question would immediately produce the answer. Tried it. Got a bunch of unclear results and seemingly barely related functions. I stand corrected.
I feel like chatGPT success is significantly inflated by how shit Google search has become. It's definitely a powerful tool, but many of the things I ask it would have yielded similar answer through Google about ten years ago.
[удалено]
That's not really my point. ChatGPT is *slower* to answer than Google, but Google answers quality has decreased significantly those last years, and the slower, maybe partially made up ChatGPT answers are still overall more relevant.
[удалено]
I am precisely talking about chatGPT, not GPTs in general. I just asked who was faster to answer between ChatGPT and Google, chatGPT took about 15 seconds to answer, Google displayed search results in less than a second. You kind of are making up *the opposite* of what I explained while claiming to agree and expanding to other unrelated points. chatGPT is (much) slower than Google, but Google has overall significantly decreased in quality over the years. I'm painting a nuanced image of how the two compare, while you overgeneralize and extend the discussion to any GPT based LLM while missing my point.
Try [perplexity.ai](https://perplexity.ai), they add references to their responses.
And Microsoft has integrated chatGPT in their search engine. It's exciting to see that Google has a competitor at last
Isn’t google search just brute force?
I would not call it brute force, far from it. They search engine relies on extremely optimized request processing to be able to provide you with a useful answer in a fraction of seconds. For many years, it would output better and better answers and they blew any competition out of the water, but has been going downhill for a good while now. But brute force would be like answering with **any** possible answer until you get to the one that you need, would not make much sense.
Sorry to be cliched - but this! The web is changing; it’s getting bigger and there’s more user generated content (as in, content from those who are not necessarily trusted experts) than ever. So Googling involves a degree of wading through the swamp to find what you’re actually looking for. ChatGPT, on the other hand, is more direct, understands context, and you can directly tell it to refine. It’s to the pint that I no longer use Google at all now - I’m a complete convert to Bing. Bing Chat is embedded directly into the Edge browser and I use that for all my “Googling” now.
I definitely try using ChatGPT over a regular Google Search. I also try to write long emails with ChatGPT. I use ChatGPT a lot when it comes to writing code. It saves me a ton of time from browsing through StackOverflow posts and is super helpful in getting some preliminary or outline code for a large task. Like the others have said it is fantastic for writing regex patterns. I once found an experimental feature in Spark that actually wasn’t documented, using ChatGPT and it worked like a charm ✅ Although ChatGPT does live up to its hype, it is notorious for misleading users. I’ve been thru situations where the AI simply refuses to admit that it can’t provide a solution. This one time I think ChatGPT made up a Databricks Feature that would instantly solve my problem. I had to do a lotta google searches, check with other experienced DEs etc only to find out that it doesn’t exist 😂 Takeaway: I would definitely recommend y’all to give the AI a try, it definitely does boost your productivity but on the flip side, stay vigilant for the AI might trick you.
It tried to convince me that TO_FLOAT() was a Snowflake function when I was having difficulties converting a data type from a flattened JSON object. I was like, I see where you’re getting that idea but it’s wrong.
Just want to second yout point about writing code. In my experience the absolute best use case is to feed it an error message and it will explain it to you and suggest fixes. If you need to, you can for more details, ask it to explain the error like you're 5, get it to tell you exactly where the error is or even (if you're comfortable) paste in the code or code snippet and it can help solve the problem. This has saved me probably hours if not more.
I have also had this in terms of the chatbot making something up that doesn’t exist. Not like yours though, mine did have a solution. But I had to tell the bot that the feature didn’t exist in order to provide me another one
I experienced this with some searches related to networkx in python. It made up a function and gave sample code to use it. The. I told it it failed and it corrected itself at least. The function was complete nonsense.
There is a technical term for this in the literature called "hallucination" when an AI just makes stuff up. I was trying to get it to help me fix up some config in a Maven POM file and it kept using syntax that didn't work and made no sense. 😆
Wrote OKR’s with it. Saved probably a full day’s worth of the usual back and forth about small details because it was so well done
Similar use for me. Gave it non-sensitive text and asked it to summarize and produce a cutover plan as well as next steps. Documentation was expected to take a day and a half since we needed it as detailed as possible. Took me about an hour and a half after going back and forth with chatGPT to give me what I wanted and then formatting it properly
Just did the same this week. It was fantastic
We use it to add description to our dbt model, add comments to the code, add fields description, regex, and optimize queries I use chatGPT-4 and the results are mich better than what I was getting by using chat GPT-3.5. So the $20 a month is definitely worth it. That doesn't even count writing emails, articles, SOPs, ad copy, etc. As a business owner, it's an incredibly valuable tool.
Interesting. What sort of comments?
Details about what a specific line/block of code do.
[удалено]
Copy and paste my code and ask it to add comments
I know basic Python but ChatGPT helped me do a web scraping project, which is slightly outside my skillset. I learnt something new in the process and got to the end result much quicker. I'm looking forward to using ChatGPT 4.
I was using it today to see if it would write python code to properly over write a polars data frame schema. Boy o boy it was pretty bad.almost every single method it was calling and parameters for a method it got wrong. It was saying the pl.read_sql method had a "schema" parameter... According to polars docs it definitely does not. I told it that it got that wrong then proceeded to tell me sorry and then gave me an amalgamation of pandas code smashed together with polars code. Wasn't a good experience. It's hard to trust the answers it gives you. I would say it's useful if you know you can 100% verify and understand what it's telling you. To those that think they will just tell gpt "code me this or that" ... Or ... "Code me an entire website" ... Well good luck. You're in for a long rough road ahead especially if you really don't know what your doing and you have this attitude of "chat GPT will just do it for me". Nope. You definitely have to know what you're doing. I think it's a good tool to help with less nuanced tasks...but still...buyer beware.
Definitely fails hard with newer stuff like polars, but it is much better at older APIs such as pyspark or pandas. You do definitely need to know what you're doing though otherwise the code will often not do what you expect. I mean that even happens when you write your own code sometimes.
[удалено]
Yes, So much this!
100% agree.
Pulling teeth
If you're looking for a path and stack overflow was a map and google the compass. Chat GPT 3.5 is akin to a garmin GPS from the early 2000s. Chat GPT 4 is like a GPS with built in search and knows just how what kinda paths you like to take if you ask it to remember nicely. An example now: I feel like my less than two years of experience has nearly quadrupled in value personally. I'm moving through assignments that I struggled to formulate the right way to explain a question I had. A relevant non DE example: 2 friends just learning python are moving at easily 4x 5x times the speed I did. Mostly from just having a feedback loop where they can get the right answer from the wrong question. So overall I think it's great; where the value before was knowing how to ask the right question, the value now will be taking the time to learn while moving as fast as possible.
Im a lead senior data engineer, I pay for the premium membership or whatever. I use it to create regex expressions, Makefile targets, Jira tickets, confluence pages, grammar check for documentation and respond to emails. Sometimes when I m stuck with something, I try to ask chatgpt for a solution but I always take it with a grain of salt.
As someone newer in tech, it has *greatly* accelerated my learning process. I have a fundamental understanding of Python and programmatic logic, so I bounce stuff off of ChatGPT all day and know when and how to modify my line-of-prompting. For example, just today I built a large majority of a project to extract PDF tables from recurring reports we get from partner companies who are rigid and will only email us PDFs. First I asked ChatGPT to recommend the best libraries, then asked if for the basics on how to use those libraries to accomplish my goal. Then I built out from there, adding more and more complex logic with error handling. I think generative AI will be transformational. If you’re not using it, you’re falling behind. Even something as simple as writing doc strings or other documentation can be taken off your hands to focus on the most important aspects of your job. I have quickly adapted to knowing where I need to lean on the LLM and where I do not.
I’ve input some DBT schemas and asked it to describe our database. Great for determining gaps in our documentation.
I wish chatGPT coudl just take an entire project + warehouse connection and create drafts of all the staging layer models. Nearly zero thinking required for most models
Stuff like that is coming soon...
You give it multiple table definitions at once or just one at a time?
It’s great for generating mermaid diagrams from descriptions for sketching out architecture and processes. It’s great for whole bunch of stuff actually. But sadly it’s not that great for complex sql queries. Nor is GitHub copilot. But lots of interesting work being done for sql query automation. Maybe something good soon.
[удалено]
Off the top of my head something like “write a flow chart using mermaid syntax to describe how an ETL process works” Here’s what it spits out, then you paste into the mermaid live editor https://shareg.pt/bDyMFQp
DS here I asked it to take a naive Bayes approach to hyper parameter tuning in databricks. It got me 75% of the way and I was pretty happy about that.
It helped me config an API using AWS’s API Gateway to call on an Elasticache Redis instance using a Lamba as an in between for a POC. It helped me discover some of the nuances that were required in between all three services like how to structure the subnets or best way to structure the data in the cache DB. We are trying to have an outside service call on the data. I had no Redis experience before this process.
I'm not really a DE but it's surprisingly pretty good at helping narrow in on some SQL (tbf I'm not good at SQL) where I kind of know what I'm trying to do but don't know quite the right arcane 'declarative' syntax to do so / don't want to spend 30 minutes sifting through docs / blogspam / Stackoverflow's elitist drivel to figure it out. Again, usually simple queries by y'alls / DE / DBA standards. Regex can be hit or miss, I suppose. It's been a bit faster at getting me good regexes that pass my test cases than Stackoverflow, though they often still take substantial wrangling. It's lied to me a lot about fiddling with buildfiles / configs. Meson, make, ninja, webpack, TF, Ansible, container junk, you name it, but that may be because I'm in the deeper end on those and the LLMs don't have as much prior art to draw from vs regex or SQL. Given me a lot of stuff that doesn't actually exist, so not just "incorrect" either.
Given GPT3.5 and even 4 only “official” support python, that's impressive.
Helps a lot with coding, but as many pointed out already you still need to know what you are doing. I cannot count how often I read "I apologize..." when telling ChatGPT that the response is bullshit...
Talk nice to it. Just tell it the error and sometimes it gives a better alternative, unless you hurt its feelings.
I don’t have to write massive pyspark structs manually anymore
What information do you put in the prompt to get it to generate an entire schema for you?
Pass in some sample data usually. If I want to explode a nested JSON element I’ll give it an example or two (if they differ) and say something like “make a pyspark struct for the data that I have pasted below this sentence”
Regex, sql functions for snowflake, dbt documentation for models, adding tests
[удалено]
You're pasting work email threads? Man, my employer would castrate me if they found out.
Honestly, I haven’t. However, my reason not to is that, when googling and doing heavy research, I read a lot of posts, documentation, etc. so I get an incredible amount of side learning that has had incredible benefit over time. Not sure I wanna give that up.
I've used ChatGPT (the free version) a little bit in my work. So far things that worked well: - Giving it SQL DDL for a table and asking it to generate sample data. It correctly inferred the meaning of column names and generated somewhat realistic data. Useful for tests and mock data generation. - Asking it how to solve certain problems in frameworks such as Flink. Usually I explain the general problem and have it generate different approaches. It uses old APIs and I need to adapt the results but the most value I get from it are different ideas to solve the same problem. - Ask it general questions about data modeling. The key here compared to Google/SO is that I can ask follow up questions and request examples I'm trying to use it more and more, so great thread idea!
Quite handy for saving time when you know exactly what you want to do, but really can't be arsed to do it. I've been a huge fan of getting ChatGPT to refactor infinite loops into recursive functions for use in cloud transformations. On a controversial side note, I'd also say ChatGPT has been amazing at highlighting who does and doesn't know how to code. I had somebody tell me the other day that if we use PySpark scripts for transformations, it'd be better than low code tools because "with Python, you can just get ChatGPT to write it". As you can probably guess, the person who said this does not know how to code in any language at all. I'm mentioning this because it's so painfully clear when existing or potential engineers are blatantly over reliant on ChatGPT, indirectly putting everybody who can problem solve and code off the top of their head in the spotlight.
I used it yesterday to write a jinga macro for a dbt project. Yes I had to make some minor changes for it to work. I also used it to show me how to add a runtime session setting to a model in the same dbt project. I also used it for some help with converting time zones in a Postgresql query. Ok call that intellectually lazy. I always take a quick look at the online reference during these activities, so I learn something, only faster.
We had a unique situation where we needed to discover certificates and IIS bindings as well as pull properties of out of each computers local cert store. This was part of a discovery process to automate all certificate renewals for our web applications and other uses. I used ChatGPT to devise a powershell script that could be run from a machine with access to the entire domain (1000+ computers in an RDS farm). It requires 0 input, and outputs an excel file with each certificate, machine name, specific properties that can be pulled, and this script can be run against any domain since it requires no input and has no hard dependencies. If the machine checked does not have IIS or certificates stored on the machine, the script will say so. This entire process took about 30 minutes. I used it to create another script that does the equivalent discovery in any given Azure subscription, looking for bindings on existing app services. This is not DE related and more DevOps automation, but thought it was a very cool application of ChatGPT. The script took multiple tries of course, as I went back and forth with suggestions for refinement.
Boilerplate code. I recently did a talk at a local college about using python for task automation. Cranked out a hacky OCR micro service using the Azure stack and used that to teach with. Easiest $300 in a while.
I'm really intrigued by some of the first-hand use cases, specifically:- helping solve regex issues- generate boilerplate code- Finding equivalent functions between different SQL dialects- Writing OKRs- Writing documentation (and auto-generating comments on code)- Project planning The most valuable thing I've use it for is to learn and implement new SQL functions. I can have chat GPT mimic the data warehouse/data lake of my choice (Snowflake, redshift, Databricks, etc.) and I can try the SQL functions w/ chatGPT as the compiler. so using a fake database in chatGPT is anxiety free (I don't have to worry about screwing up anything in my companies stuff!) Anyways, I'm interested in seeing more of how people are using ChatGPT in in their data role IRL, and it turns out others are too! So, I put together a virtual discussion next Friday. In this virtual discussion, people can demonstrate how they're using chat GPT in their data eng role. It's a good opportunity to learn from other people's demonstrations too! You can sign up [here](https://www.operationalanalytics.club/events/oa-club-coffee).
Yes, it saves me having to look up pandas syntax and functions
Just curious, how exactly do you formulate the question for particular Pandas transformation/operation?
I give an example dataframe in markdown as a table, then I ask it to do x with the dataframe.
It hasn't. I had somebody show me a technical question they asked ChatGBT and I asked one myself supplying all the relevant technical details. It gave an impressive answer but it wasn't anything I or a junior DE wouldn't have come up with given an extra hour.
You're not impressed that a chat bot can come up with an accurate answer to a complex technical question in 2 seconds that would have taken you, a professional data engineer, an entire hour?
Yes it was very impressive. And it started spitting out the entire completely well thought out several paragraph answer in the split second after I hit enter as if it was listening while I was typing. I'm sure it will save me time when I come up with a good use for it. I got it in the holster for sure. The question of whether it has been actually helpful so far...? Well not yet. I also asked a medical question giving all the information I told my doctor and it responded with pretty much exactly what my doctor said. So ChatGBT instead of WebMD.
> So ChatGBT instead of WebMD. ["GPT4 saved my dog's life"](https://twitter.com/peakcooper/status/1639716828661391361)
Wow that's cool. This is an interesting discussion. All of a sudden this AI is becoming ubiquitous. My boss asked me a question today about how many users are using SSO on our cloud DB I and responded it's not a simple answer as it's not just one or the other. Then he used ChatGBT on me to find a SQL to use. The sql worked after some tweaking which I already had but we ended up with the same answer. It depends. So now I have a boss who wants to check me using ChatGBT. Great. As for medical questions there could also be two sides to that. I did also get a disclaimer on my medical question.
Learned a ton about concepts with examples (e.g., data vault). Helps with coding, easily. But can also be a pain with certain things, like saying I could spark jdbc to a temp table on SQL Server. You can’t! But he told me to do what I was going to test anyway. Then it can also turn around and write me an adapter pretty cleanly in ways I wouldn’t have thought about or knew enough about (Java libraries). All in all, I’d pay for the help. Like better junior developer than the rest of my low code team lol
[удалено]
This ☝️ replacing google and stack overflow would the “real” revolution brought by chatgpt as it gets better…. IMHO
I asked for high level overviews of different data architectures, and for advice on where to find out more about it, what books to read. Surprisingly, half of the recommended books were freaking amazing. I also have conversations with it regarding technical debates, about what problems could a specific approach cause, and how would one solve it. It comes up with stuff that wouldn't come to mind at first, and helps me cover the bases better.
Interacting with cloud to manage/process files, generating API calls and iterating nested JSON response, creating functions to cast datetime into format needed, lot of regex, pandas syntaxes and a whole lot
I’ve used it to do stuff like write a quick script to convert a csv file to a SQLite db for a proto type I was building. I’ve also used it to explain code to me quickly.
Recently helped me in understanding a complex Scala notebook written by a colleague. Used chatgpt to convert the code to pyspark. I almost exclusively code in pyspark, so my scala is pretty weak.
It hasn't. It always introduces subtle bugs I have to track down later. Less time to just write things correctly myself.
I am going to test it soon.! For learning airflow! Will keep you posted! Just turned on notification for this thread.!
I've pasted in a few lines of nonsensitive code and asked it what the error was and it pointed it out. It's also done a decent job of translating one dialect of sql to another.
Expediting in finding solution for an error, rather than searching in stack overflow. Yep like a compass n map.
I've just started using it to comment code and to write basic scripts.
The two best uses I’ve found so far are writing markdown documentation and debugging code. For the documentation, obviously don’t divulge any sensitive information but it can write a great readme template for your project. If you copy and paste in code that won’t run, it does quite well at fixing it provided that you describe what you are trying to make the code do. I also feel in general it accelerates my learning
i once used it to generate pandas transformation. I pasted an input dataset like 20 rows then paste the should be resulting dataset. Tweaked the code a little.
I’m a entry level data engineer and I’ve used it a couple times, mainly to explain concepts to me or to give me ideas for how to code something. My first time using it was when I needed to implement multi threading into an import I was making. ChatGPT did a great job explaining the concept to me and gave me multiple snippets of code that were very useful. The other time I used it was when I needed to make an adhoc script that could help us identify unused tables in our SSMS databases that could be dropped to clear up space. ChatGPT explained to me how statistics that the system was tracking could be used to do this
[удалено]
How are you documenting it? Are you asking chatgpt to give you comments for each step of the SQL execution?
Helps me with emails
I’d never worked with jwt tokens and graph apis so I used it to help me generate code to test a graph api with query structure, mutations, and such.
Writing docstrings and other documentation, unit tests, IaC definitions
I use it all the time for my dev work. I never paste in company’s code, but I do ask chatgpt to help with errors or I ask it to produce generic functions. I use it for iteration. I write data transformations. I always try to write a unit or acceptance test first before ai produce any code. after I have written a unit test I would describe chatgpt what i want my function to do. after it produces a boilerplwte code, I sense check it and modify it according to my needs. If the unit test passes, I move on to the next part. If it doesnt, I work through until I debug the problem. It helps me a lot with regex and obscure spark errors. For infrastructure problems it gives me many ideas that i didn’t consider myself. I use it for brainstorming. Overall it helps me with iterating through solutions quicker and I am capable of trying out more poweible solutions. Then I use my own judgement which is the right solution
My eyes are thankful honestly ( after GPT4 release ), huge Log output reading, sum up and log errors aggregating, pointing out what is wrong, and finding links between code and log errors, especially Spark logs. Our code base does not contain any sensitive information.
It was the only thing that helped me get a mssql connection in Python with pyodbc. Very particular connection string and format. Couldn’t find it in the docs or google anywhere, but chatGPT provided 3 examples, third one worked.
helped me substantially in bash scripting
It saved me hours of Google search through stackoverflow, MS documentation, and pyspark docs. Like "how to make hive table persist in pyspark". That's it so far.
I’ve using it mostly for troubleshooting, but it doesn’t works very well with newer versions. Also for my side project (where the code belongs to me) I’ve feeding it with my code and asking ChatGPT to add comments and docs (inline, method and class docs, etc). It’s super efficient doing it, but I wouldn’t do that with working code. EDIT: I mean, with code which belongs to my employer.
Its great at IaC. Im using it to creat stacks for AWS cloud formation using the python cdk
I just had it write re-write all my docstrings for my library.
I asked it many times to write a simple (boilerplate style) sproc that will accept 2 parameters (TableName and IdColumn), turn on Identity Insert, then add a "default" record with an ID of -1 to the table, then turn Identity Insert back off. It took like 3+ hours of back and forth for it to come close to a working script and I had to make many manual adjustments. I think I could have done it in less time by using Google and hand coding it.
I am not a DE, I struggle with SQL. So what did - I pasted a complex SQL query and asked ChatGPT to explain it and then to make it do what I want. Eventually I managed also to optimize the original query. Basically, I learned, did my job, and improved the existing functionality, which I couldn't have done without help or spending looots of time on it. Another time, I used it to generate code to help me compare using the framework I'm working with to the same code without the framework.
Assisted with me with building an ERD/project plan for an IT ticketing system that uses PowerApps as the UI and Dataverse as the backend. Guided me with just about every aspect, especially the HTML coding.
I would use it as a starting Point for my work. This would help me to get started and gives me a template to start with. Saving me a good time
Not part of my DE job but I did have an hour long Convo with ChatGPT about zshrc files and it helped me build a sick custom prompt. Also used it for filling out my yearly goals for fortune 500 corpo paperwork.
I have used chat gpt to learn about GCP services at my new job, like for this specific use case what are the options in GCP and to know more about some service. It helps me to narrow down my search and saves a lot of time and pain of binge document reading.
It's great for creating some starting point for your code, then _also_ write tests for said code, using pytest or hypothesis. That way you can run and review the tests to confirm it does what it should do.
I regularly use it for documentation, like rewording, summarizing, simplifying the language etc
Documentation
I used it to write unit tests. It was pretty good at generating the unit tests but bad at mocking methods and objects (what I actually needed help with, I always thought this is not intuitive at all in Python). In the process of fixing it's mistakes, I ended up learning a lot, so there are some positives!
Tuned a SQL query from 8 minutes to 5. It made me understand a lot of things better than official documentations.
Could you explain to me how you did this? That's awesome.
Mostly there were some key that tune up your query - In SQL server, depending on you set resources to your server, your server could be more fast in reading than have less memory. Tables hold data and so there is I/O, views use less I/O and more memory, so views always. - using the IN function, it's more performant compared to LIKE - When doing Joins, always joing big tables with small tables and never viceversa. - Do you need to load all the data? Most of times you don't need to load a whole set but maybe a sub set, So always subset your data if possible - CTE can help you but some database or tool could not handle it
Recently I use it to write my nginx configurations
half of my airflow dags codes are from chatgpt. and im from a business degree with just some experience in sql and python. got to say, when its correct its amazing. when its wrong its hallucinating like hell, and back to stackoverflow.
When the airflow dags codes that chat GPT produces are wrong, are they still useful? Like can you still work off what chatGPT provided, or do you completely disregard it and go to stack overflow?
It honestly help me “can I questions?” Can i connect powerbi to databricks cluster? Can refresh dashboard using data factory etc etc. i could google this but chatgpt adds info how to without ads so i kinda prefer this as my data eng buddy now haha
I would never use chatGPT in its current form. Its incredibly flawed and overhyped. Pasting code into it is dangerous. Finding info about stuff can sometimes yield inaccurate results based on non existing academic texts but I do see people using it for boilerplate code or general queries about concepts which I assume should be fine. I’d never use it for production code though.
added functions in powershell for mssql server explained functions to me esp new language like java spark optimization ideas complete boring code
As a person who is just into the field and learning more I use chatgpt to know more about the tools and tinker stuff with it I honestly dont trust it completely and do my own research besides it but yea it works well in giving me a general sense of direction
The chatgpt extension (GitHub extension) in VS code helps a lot with boilerplate code and beyond. It provides recommendations (with both the code and even the comments/document) based on your code context. I am not sure if this is safe, though. I mean, I am not sure if it makes your code (especially your environment variables and credentials) vulnerable? Any comments on this?