TWO KEY INITIATIVE WERE:
INITIATIVE 1
Data Architecture
INITIATIVE 2
Credit Card Loan Validations
INITIATIVE 1
Data Architecture
INITIATIVE 2
Credit Card Loan Validations
Data Architecture
We were involved in defining a new, scalable data architecture (Medallion on AWS stack with Airflow/dbt) to replace manual, obsolete processes.
Credit Card Loan Validations
This meant building an in-house microservice using a Drools-based rules engine, integrated with Data Architecture, to replace a costly third-party vendor.
The outcome was a new, scalable data platform with established best practices, reduced annual vendor expenses by over $1M, and increased organizational adoption, positioning our team as an internal “platform team” to guide future innovation.
CLIENT & CONTEXT BACKGROUND
General Context
In conjunction with one of our American partners, we have worked for one of the largest and oldest consumer financial institutions in the United States. Their core business is to offer accessible personal loans and insurance.
As a large, highly digitized company, they have a strong cross-functional engineering team focused on maintaining core-business applications and an innovation team focused on new initiatives. Innovation seeks to generate a mapping between the current state and the target state, which involves an architecture, a goal, and a responsible team.
LoopStudio Team Context
We embedded a strong, high-performing squad in the Innovation team of the client. One of our senior leaders took on the role of Principal Architect, leading various initiatives. Some of them related purely to data, others to the use of that data.
We formed a team with our Principal Engineer, CTO, Tech Lead, and a Sr. Developer. They took the roles of Engineering Manager, Tech Lead, and Sr. Developer, respectively, and we joined one of the existing teams composed of about 30 people, and responsible for 5 to 6 initiatives. Collaboration with other teams and units was critical to the success of the initiatives.
Initiative 1
Data Architecture
BUSINESS PROBLEM
The organization’s data processes are very manual, using obsolete technologies for which the teams are not trained. Generating ELT pipelines for multiple consumers that scale is a pain for the business.
Solution & Approach
The Principal Architect defined the vision for the new architecture, a transversal initiative that feeds the target state and innovation projects with relevant data.
This involved some key technologies in which to train other teams in the future, based mainly on the AWS stack:
Orchestrator
Apache Managed Airflow. Used for orchestrating multiple DAGs in parallel, handling specific logic with Python code, and its simple integrations with the entire AWS ecosystem and data lakes.
Modeling
dbt. Used to build reusable models for a Medallion architecture (raw, intermediate, and marts models), which include tests and assertions for data quality, are self-documented, and hosted on GitHub.
Storage
S3, Aurora, and Snowflake. S3 as cold storage and raw data; Aurora as an intermediate/silver layer; and finally Snowflake for analysts and the data science team.
Event driven
AWS SQS, AWS Lambdas. Used to facilitate data integration from certain current state systems where Airflow could not connect, allowing legacy systems to push to SQS for consumption by Lambdas.
Technical Challenges & Overcoming Them
Multiple and heterogeneous sources
Involved generating multiple connectors and reusable DAGs (e.g., those consuming from S3) to unify data from operational systems, S3, spreadsheets, and Oracle DBs. Defined the scale of the Airflow instance to respect parallelism limits.
Modeling
Extensive modeling work, defining core fundamental tables vs. specific marts in the data warehouse. Key to this was understanding data sources and holding bi-weekly sessions with stakeholders and architects to achieve an ERM (Entity-Relationship Model).
Best practices from scratch
Defined best practices for Airflow and dbt, built Data Architecture with templates and example DAGs, and documented them in Confluence. The team took charge of code review to ensure adherence to conventions, allowing the team to function as a ‘platform team’ and foster collaboration.
Data volume & Near Real-Time
Processed around 50 thousand applications per day. Used dbt incremental models and Airflow checks to implement an incremental pipeline (not full refresh). Managed to process data every 5 minutes to get closer to ‘near real-time’. Decoupled highly nested JSON data in the transformation layer for optimal queries and performance.
Data Pipelines best practices
Implemented key fundamentals: Observability (centralized logs in CloudWatch and alerts in Slack), Data integrity tests with dbt, small decoupled tasks, Backfilling by date range, and Idempotence (using transactions in the Medallion schema to prevent duplicates during backfilling).
Managerial/Modeling Scalability
The Engineering Manager/Lead managed stakeholders to define the data model. The main challenge was integrating new initiatives without affecting the generality of the solution. This was addressed by heavily separating the core, generic, and fundamental model from the specific marts for use cases.
Results & Impacts
Performance
Managed to execute 10 critical DAGs for the organization incrementally with execution times between 3 and 15 minutes.
Business Enablement
Generated data marts for the data science, vehicle loans, and credit cards teams, enabling them to make better decisions.
Resilience
Enabled safe performance of backfillings without duplicating data in the event of data errors, code errors, or upstream changes.
Adoption
Due to the success in utilization and adoption by interested stakeholders, Data Architecture became the de facto platform for initiatives requiring key data on loans and credit information.
Initiative 2
Credit Card Loan Validations
BUSINESS PROBLEM
The client was paying $1 million a year for a provider that, based on information from the loan creditor, determined whether the loan could be granted or not and what the reasons were. For example: age, salary, geographic location, outstanding debts, etc.
Upper management defined the goal of developing this in-house. The final objective was to remove the dependence on the third party and thus optimize costs.
Solution & Approach
Our Engineering Manager worked closely with the credit cards team to understand their current system architecture and expectations for interacting with the new microservice.
In parallel, the team developed a rules engine system (using Drools, an open source tool), which allowed for visually or programmatically generating complex business rules and their reasons for loan rejection.
A microservice in Java was developed, designed to be highly resilient to cards traffic. This service would:
Phase 1
Take the relevant information sent by cards
Phase 2
Hydrate with information from the new data solution
Phase 3
Send it to the rules engine
Phase 4
Give a result
Technical Challenges & Overcoming them
Understanding the credit cards business
Achieved through knowledge-sharing and whiteboarding meetings. The team created example rules and a template repository, making the cards team participate and performing code review on Drools logic, enabling the team to generate rules autonomously.
Passage to production and testing
A critical service that could generate erroneous loans.
The team used synthetic data and past cases for testing.
A gradual strategy was suggested using a feature flag to pass traffic from 1% to 100% day by day.
Generating a scalable model / Delegation
Post-production, we transitioned to a consulting/architect role to delegate the project. Provided KT sessions, support on production issues, support during early morning releases, and extensive code review to ensure the cards team could continue adding features while respecting the original architecture and tech stack designed to be part of a platform team.
Reusing the Data Architecture and response times
Required adapting the Data Architecture for low latency. Implemented a cache layer with the DevOps team, saving the ‘features’ for the rules in an instance for faster reading. Included a custom library in the microservice for cache management to update information across different tiers (hot, warm, cold – Redis, Aurora, S3).
Results & Impact
Economy
The final objective was achieved, cutting high costs for the organization (a $1M annual expense) and removing dependence on a vendor, making it one of the first successful cases of in-house development in this stage of innovation.
Collaboration & Leadership
The success was noted by upper management, leading to the promotion of the idea that our team should define guidelines, POCs, and initial architecture to guide others, solidifying a pattern of internal platform team mentorship.
If you are looking for a Tech Partner to help your startup reach the next level, contact us below, and let’s get started!