I was in fintech for the last 7 years and recently changed my domain. It was great. It is hard to give very specific advice but here is what I can say.
1. Fintech is powerful and most of the times you have huge web traffic. Even otherwise, one big mistake can be fatal because it is regulated and in the age of social media, people use it if they find anything wrong.
2. Due to the above, it is very important for your team to understand some financial basics. Before you begin any development work, I would highly encourage sessions where the team does explorations with domain experts on learning the same. You might want to consider a sprint or two to do this.
3. Banking regulations will pose deadlines too, which are usually non-negotiable because the government sets them. So, while agile doesn't like the concept of deadlines, this is where we need to bring in the risk management hat of waterfall. Make risks visible to your management and document it so that people know. Get your PO to prioritize (which could be making hard decisions to drop requirements) should such a situation arise.
4. We did a lot of MVP i.e. minimum viable product planning in order to figure out the number of sprints it would take us to deliver a feature. It helped the team to understand the big picture and drove better conversations leading to better requirements. Your estimates will improve over time but this is important so that you don't commit to an unreasonable date.
5. Always have cards that capture your production and post-production monitoring efforts because a customer complaint could mean that you might have to put in a hotfix or rollback a release. Invest in the due diligence part and you would see better results.
6. Due to the sheer volume of traffic, fintech industry especially is always on the forefront of technology, which means that spikes, hackathons etc. would be necessary for exploring technology.
7. Other than the above, it is all standard practice that you see in the books. But, again you would have to figure out what works for your team and tweak it based on the organization you are in. Hope this helps.